A Unity Catalog migration is not a table copy. It is an identity migration, a storage migration, a permissions migration and a workload migration, and they have to happen in that order. We run it in waves: assess everything with automation, move groups to the account, replace mounts with governed external locations, upgrade tables by type (sync external tables in place, copy managed ones), move jobs and clusters to Unity Catalog access modes, then cut over domain by domain with a rollback path. Below is the sequence, the before-and-after permission model, the decision table we use per table, and the failure modes that cause most of the pain.
Why now, and why it keeps getting postponed
Unity Catalog is where Databricks is putting all its new capability: lineage, system tables, attribute-based access control, Lakeflow, serverless compute, Genie, Model Serving governance, AI agents. Workspaces still on the legacy Hive metastore get less of it every release, and the older mechanisms they depend on — table ACLs, credential passthrough, DBFS mounts — are deprecated or on their way out.
And yet the migration keeps slipping, for a sensible reason: production runs on what is there today. Nobody wants to be the person whose migration broke the month-end close. The answer is not to be braver. It is to make the migration boring — sequenced, measured, and reversible at every step.
What usually triggers the project: an AI use case. Teams discover that Genie, vector search, Model Serving governance and agent tooling all assume Unity Catalog, and the migration they deferred is now on the critical path of something the business wants.
Why it is not a table copy
The common mental model is “move the tables from hive_metastore into a catalog.” Tables are actually the third thing you move. In practice there are four migrations hiding inside one:
| Migration | What changes | What breaks if you skip it |
|---|---|---|
| Identity | Workspace-local groups become account groups synced from the IdP | Grants cannot be expressed in Unity Catalog at all |
| Storage | Mounts and instance profiles become storage credentials and external locations | Tables migrate, but jobs still read files around the governance |
| Data & permissions | Tables, views and functions move into catalogs; legacy ACLs become UC grants | Users lose access, or get too much of it |
| Workloads | Jobs, clusters, notebooks, SQL and dashboards point at three-level names on UC-enabled compute | Everything looks migrated, but production still runs on the old metastore |
Six waves, each with an exit gate
We run the migration as overlapping waves. Each wave has an exit gate, and each domain passes through the waves on its own schedule, so the first domain can be in production on Unity Catalog while the last is still being assessed.
Wave 0 — Assess with automation, not interviews
Inventory everything: workspaces, tables by type and location, views, UDFs, grants, groups, mounts, clusters, jobs, notebooks, init scripts, service principals. Databricks Labs’ UCX toolkit does much of this, and we add a dependency map that links each table to the jobs that write it and the jobs, dashboards and extracts that read it. The output is a wave plan by domain and a risk register. Exit gate: every production table has an owner, a target name and a migration method.
Wave 1 — Identity
Set up SCIM from your IdP to the Databricks account, create account-level groups, and migrate workspace-local group memberships and entitlements. Create service principals for every production job. Exit gate: no grant in the target design refers to an individual or a workspace-local group.
Wave 2 — Storage
Create storage credentials and external locations for every path the inventory found, and map each mount to the external location that replaces it. Exit gate: every mount has a governed replacement, and the grants on it match who actually needs file access.
Wave 3 — Tables, views and grants
Create the target catalogs and schemas, upgrade tables by type (the decision table is below), recreate views with three-level names, and translate legacy ACLs into Unity Catalog grants on account groups. Exit gate: row counts and checksums match for every migrated table, and an access matrix test shows the right groups can — and the wrong groups cannot — read each object.
Wave 4 — Workloads
Move jobs to UC-enabled compute (standard or dedicated access mode), replace /mnt/ paths with volumes or external locations, change two-level names to three-level names, and run jobs as service principals. Exit gate: each job has run in parallel against Unity Catalog and produced identical output.
Wave 5 — Cutover and decommission
Switch writers, then readers, domain by domain. Make the legacy copies read-only, then retire them after an agreed soak period. Exit gate: zero reads on the legacy objects for the soak window, verified from audit logs.
Identity first, because grants depend on it
Identity is the least glamorous wave and the one teams most want to skip. It is also the one that decides whether everything after it works. Unity Catalog grants are made to account-level principals. If your groups still live inside individual workspaces, there is nothing to grant to.
- Reconcile before you migrate. Workspaces often have groups with the same name and different members. Decide which one is right before creating the account group, or you will merge two different meanings of “analysts”.
- Service principals for every production job. Jobs that run as a person break the day that person leaves, and their access is usually far broader than the job needs.
- Turn off personal access tokens for production. Move integrations to OAuth with service principals during the migration, not after it.
The permission model, before and after
The reason the migration is worth doing is visible in one picture. On the legacy metastore, access is decided in several places that do not know about each other. In Unity Catalog there is one place, and every path to the data passes through it.
Translating legacy table ACLs is mostly mechanical. The hard part is the access that was never written down — the analyst who read a table through a mount for years, the job whose instance profile could see the whole bucket. That access is lost when the mounts go, which is the right outcome, but it needs to be discovered in Wave 0 and granted explicitly, not discovered in production on cutover day.
-- Grants go to account groups, at the highest level that is still correct GRANT USE CATALOG ON CATALOG claims_prod TO `claims-readers`; GRANT USE SCHEMA, SELECT ON SCHEMA claims_prod.gold TO `claims-readers`; GRANT MODIFY ON SCHEMA claims_prod.gold TO `sp-claims-pipelines`; GRANT READ FILES ON EXTERNAL LOCATION claims_landing TO `sp-claims-ingest`; -- Nothing granted on silver to anyone outside the domain
Which method for which table
Not every table moves the same way. This is the decision table we apply to each row of the inventory.
| Table type | Method | Notes |
|---|---|---|
| External Delta table on cloud storage | SYNC to a UC external table | Metadata-only, no data movement; fastest path |
| Managed HMS table on DBFS root | DEEP CLONE or CTAS into a UC managed table | Data must move; DBFS root is not governable |
| Non-Delta external table (Parquet, CSV) | Convert to Delta, or keep external with a plan to convert | Convert where a writer can be changed in the same wave |
| Very large or hot table | Clone, then incremental re-sync until cutover | Keeps the cutover window to minutes |
| View | Recreate with three-level names | Migrate after every table it references |
| Hive UDF | Rewrite as a SQL or Python UC function | Often the long tail of the project |
| Unused for 90+ days | Do not migrate; archive | Verify with audit logs and query history, not opinions |
Hive metastore federation can buy time. Where a domain cannot move yet, federating the legacy metastore into Unity Catalog lets you govern and audit those tables from UC while the full migration is scheduled. We use it as a bridge, not a destination.
Workloads: where most of the effort really goes
In most estates we assess, tables are a minority of the effort. Code is the majority. Notebooks with hard-coded /mnt/ paths, two-level table names, RDD APIs and init scripts that are not allowed on standard access mode, JDBC tools pointed at a workspace URL, and dashboards nobody remembers creating.
- Automate the code scan. Lint every notebook and repository for mounts, two-level names, deprecated APIs and access-mode incompatibilities, and attach the findings to the wave plan.
- Parameterize the catalog. Code that reads
${catalog}.gold.claimmoves between dev, staging and prod without edits — and makes the next migration trivial. - Run in parallel before you switch. Every production job runs against Unity Catalog alongside the legacy run, and outputs are compared, for at least one full business cycle.
Cutover patterns and rollback
We use three patterns, chosen per domain:
| Pattern | When | Rollback |
|---|---|---|
| In-place sync | External tables whose writers are already on UC-enabled compute | Writers point back at the legacy table; data is the same files |
| Parallel run, then switch | Managed tables and critical pipelines | Legacy pipeline kept warm and read-only until the soak ends |
| Freeze and move | Small, low-change domains | Restore from the pre-cutover clone |
The rule in every pattern: switch writers before readers, and never let both metastores accept writes for the same table. Dual writes are how data gets silently forked, and silently forked data is the one failure that is genuinely hard to undo.
The failure modes we plan for
1. Migrating tables before identity. Tables land in Unity Catalog with nobody able to read them, and the team grants access to individuals to unblock people. Those grants become permanent.
2. Leaving the mounts in place. Everything appears migrated, but jobs still read through /mnt/, so governance and lineage are bypassed and nobody notices for months.
3. Underestimating code. Teams plan against the number of tables when the real work is the number of notebooks, jobs and downstream tools.
4. Recreating the old layout. One catalog per legacy workspace looks fast, but it carries all the old sprawl into the new system. The migration is the cheapest moment you will ever have to adopt a proper catalog design — domain plus environment, as in our reference architecture.
5. No decommission date. Without a date, the legacy metastore stays “just in case” and the organization pays for two systems and governs neither properly.
Readiness checklist
| Before you start | Ready when |
|---|---|
| Inventory | Every table, view, function, mount, job and grant is listed with an owner |
| Dependency map | Each production table links to its writers and readers |
| Target design | Catalog, schema and naming conventions are agreed and documented |
| Identity | SCIM is live and account groups are reconciled |
| Compute | Runtime versions and access modes are chosen for every job |
| Validation | Row-count, checksum and access-matrix tests are automated |
| Rollback | Each domain has a written rollback path and an owner |
| Decommission | A date is set for making legacy objects read-only, and another for retiring them |
The bottom line
A Unity Catalog migration breaks production when it is treated as a data copy. It goes smoothly when it is treated as what it is: identity, then storage, then data and permissions, then workloads — in waves, domain by domain, with an exit gate on each wave and a rollback path on each cutover.
Done that way, the migration also becomes the cheapest moment to fix the catalog design, retire unused data, move jobs to service principals and remove every mount. The organizations that use it that way come out with a platform that is ready for AI. The ones that just copy tables come out with the old sprawl under a new name.