Unity Catalog in Practice: Metastore Layout, Grants That Scale, and Leaving hive_metastore

Unity Catalog is the least optional part of modern Databricks. Every feature that matters shipped in the last three years — lineage, Delta Sharing, liquid clustering defaults, predictive optimization, serverless — assumes your tables live in it. Staying on hive_metastore is choosing to run 2021 Databricks forever.

But UC punishes improvisation. The three-level namespace looks trivial in the docs, and then you meet a workspace with catalogs named test, test2, dinesh_dev, and prod_final, each with its own grant archaeology. Governance debt compounds faster than code debt because nobody refactors permissions for fun.

This is the layout I set up on day one, the grant model that survives team growth, what lineage actually captures, and the migration path off hive_metastore that does not stall in month three.

The namespace: decide once, early

Unity Catalog gives you catalog.schema.table. One metastore per region, attached to every workspace in that region. The metastore is plumbing; the design decision is what a catalog means.

After trying both, I use environment-first catalogs for small and mid-size teams, and domain catalogs only when domains have separate owning teams:

flowchart TD
  ms["Metastore: one per region"] --> dev[Catalog dev]
  ms --> stg[Catalog staging]
  ms --> prd[Catalog prod]
  prd --> bronze["Schema bronze: raw ingested"]
  prd --> silver["Schema silver: cleaned, conformed"]
  prd --> gold["Schema gold: business marts"]
  bronze --> t1["prod.bronze.orders_raw"]
  silver --> t2["prod.silver.orders"]
  gold --> t3["prod.gold.revenue_daily"]

Environment catalogs, medallion schemas. Boring, and boring is the point.

The medallion split maps cleanly onto schemas because grants follow layers: analysts read gold, engineers read silver, almost nobody reads bronze. That grant story is why I do not use one catalog with schemas like dev_bronze — you cannot express “analysts see prod gold only” cleanly when environments and layers share a level. The layer definitions themselves are in the lakehouse guide.

Storage: each catalog gets its own external location so blobs are segregated per environment, and I default to managed tables inside it. Managed tables let Databricks own layout, run predictive optimization, and handle compaction without a scheduled job. External tables are reserved for data that another engine writes or that must survive a Databricks exit — the same boundary question as in Delta vs Iceberg.

Grants that scale past ten people

Every UC permission mess I have audited had the same root cause: grants to individual users. People change teams. Groups do not.

Rules I enforce in review:

• Groups come from the identity provider via SCIM. No workspace-local groups for data access.
• Grants go on catalogs and schemas, never on individual tables unless a specific table is genuinely more sensitive.
• Service principals own production. Humans do not write to prod gold from a notebook.

The grant set for a typical team:

-- Analysts: read gold, nothing else
GRANT USE CATALOG ON CATALOG prod TO `grp-analysts`;
GRANT USE SCHEMA  ON SCHEMA prod.gold TO `grp-analysts`;
GRANT SELECT      ON SCHEMA prod.gold TO `grp-analysts`;

-- Data engineers: read everything, write silver and gold
GRANT USE CATALOG ON CATALOG prod TO `grp-data-eng`;
GRANT USE SCHEMA, SELECT ON SCHEMA prod.bronze TO `grp-data-eng`;
GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE
  ON SCHEMA prod.silver TO `grp-data-eng`;
GRANT USE SCHEMA, SELECT, MODIFY, CREATE TABLE
  ON SCHEMA prod.gold TO `grp-data-eng`;

-- The pipeline identity that actually writes prod
GRANT USE CATALOG ON CATALOG prod TO `sp-etl-prod`;
GRANT ALL PRIVILEGES ON SCHEMA prod.bronze TO `sp-etl-prod`;
GRANT ALL PRIVILEGES ON SCHEMA prod.silver TO `sp-etl-prod`;
GRANT ALL PRIVILEGES ON SCHEMA prod.gold  TO `sp-etl-prod`;

Two things trip everyone. First, USE CATALOG and USE SCHEMA are prerequisites, not permissions in themselves — SELECT without them resolves to “table does not exist,” and you will debug a phantom missing table for an hour. Second, GRANT ... ON CATALOG with SELECT cascades to every current and future schema; convenient for engineers, almost always wrong for analysts.

Row filters and column masks handle the finer cuts — masking PII columns from analysts while engineers see through — and they belong in the same PR as the pipeline that adds the column. If you publish schemas to consumers, treat grants as part of the interface, the same way data contracts treat schema.

Audit quarterly with a query, not by clicking:

SELECT grantee, privilege_type, table_catalog, table_schema
FROM system.information_schema.schema_privileges
WHERE table_catalog = 'prod'
ORDER BY grantee, table_schema;

Lineage: automatic, with a blind spot

UC captures lineage automatically for anything executed on UC-enabled compute: table-to-table and column-to-column, across notebooks, jobs, DLT, and SQL warehouses. No agents, no dbt docs sync, no scraping. When an analyst asks “where does revenue_daily.net_amount come from,” the answer is two clicks in Catalog Explorer or a query against system.access.table_lineage.

The blind spot: lineage only sees UC-governed workloads. A job on a legacy cluster with no UC access mode, a Kafka producer writing files that Auto Loader picks up, an external engine writing an external table — invisible. Lineage tells you what Databricks did, not what happened to the data. I treat it as the 80-percent map and keep source-system documentation for the edges.

It is also retention-limited (about a year of history). For “what built this table in 2024,” you still want code in git.

Migrating off hive_metastore

The migration is not the hard part; the long tail of jobs with hardcoded two-level names is. My playbook:

1. Stand up the UC namespace and grants first. Empty catalogs, correct permissions, external locations registered.
2. Inventory hive_metastore — every table, its format, its writers. Databricks’ assessment tooling (UCX) does this well.
3. Migrate external Delta tables with SYNC, which registers them in UC without moving data. Managed hive tables need a deep clone or CTAS into UC-managed storage.
4. Repoint readers first, then writers, job by job.
5. Freeze hive_metastore read-only. Delete it a quarter later.

-- External Delta table: register in UC, no data movement
SYNC TABLE prod.silver.orders
  FROM hive_metastore.silver.orders;

-- Managed hive table: copy into UC-managed storage
CREATE TABLE prod.silver.customers
  DEEP CLONE hive_metastore.silver.customers;

The repointing step is where big-bang plans die. Do it per pipeline, in the pipeline’s own repo, behind its own PR. Default catalogs help during transition — set the workspace default catalog so unqualified names resolve to UC — but I still make every production job use fully qualified three-level names. Implicit resolution is how a dev job writes to prod.

flowchart LR
  hm["hive_metastore.silver.orders"] --> sync["SYNC or DEEP CLONE"]
  sync --> uc["prod.silver.orders"]
  readers[Readers repointed] --> uc
  writers[Writers repointed] --> uc
  hm --> ro["hive_metastore frozen read-only"]

Readers move before writers. The old namespace dies read-only, not deleted.

Streaming jobs deserve care: a structured streaming write repointed to a new table path needs its checkpoint story sorted first, or you will reprocess or drop data. That is checkpoint discipline, covered in the Structured Streaming post.

Pitfalls

Catalog sprawl. If anyone can CREATE CATALOG, you get namespace slums within a month. Restrict it to admins and publish the naming convention in the repo.

Granting to users. Works fine until the third re-org. Groups from the IdP, always, even for the two-person startup phase.

Forgetting USE privileges. SELECT without USE CATALOG and USE SCHEMA reports the object as nonexistent. Bake all three into every grant script.

Humans writing prod. Notebook-to-prod-gold is a governance model with a hole in it. Production writes go through service principals in jobs, code in git.

SYNC on tables another engine still writes. SYNC registers the current state; a non-UC writer afterwards causes drift between metastores. Migrate the writer in the same change.

Trusting lineage as compliance evidence. It misses non-UC workloads. Great for debugging and impact analysis; not a complete audit trail on its own.

FAQ

One metastore or one per environment? One per region. Environments are catalogs inside it. Separate metastores per environment resurrect the cross-workspace sharing problem UC exists to solve.

Managed or external tables by default? Managed, under UC. You keep Delta format and full external readability of the files, and gain predictive optimization plus lifecycle management. External is for tables other engines own or write.

Does Unity Catalog work with Iceberg? Yes — UC can serve tables over the Iceberg REST catalog protocol, and Delta tables can expose Iceberg metadata via UniForm. The catalog-strategy decision is bigger than Databricks; I walk the options in Iceberg catalogs.

How granular should grants get? Schema-level for 90 percent of cases. Table-level grants and row filters are for genuinely sensitive data, not the default — every exceptional grant is a future audit finding.

Is UCX mandatory for migration? No, but its assessment scan is the fastest inventory of what will break. I use the scan output as the migration backlog even when I run the actual moves by hand.

What this means for data engineers

Unity Catalog is infrastructure, and infrastructure rewards the boring decisions made early: one metastore, environment catalogs, medallion schemas, group grants, service principals for prod. Written down, enforced in review, automated in Terraform if you have it.

The migration from hive_metastore is a grind, not a risk — SYNC and clones are safe, and the work is repointing jobs one PR at a time. Every quarter you defer it, the inventory grows.

Do the namespace design this week. Grants to groups next. Lineage and the fancy features arrive on their own once the tables live in the right place.