The Governed Lakehouse: An Enterprise Reference Architecture on Databricks

Why a reference architecture, and not just more workspaces

Databricks makes it easy to start. A team spins up a workspace, connects a bucket, writes some notebooks, and ships a dashboard. Then another team does the same, and another. Two years later the enterprise has eleven workspaces, four ways of naming tables, three copies of the customer table, and nobody who can say with confidence who has access to what.

None of those decisions were wrong at the time. The problem is that each was made locally. A reference architecture is simply the set of decisions you make once, centrally, so that every new team inherits them instead of reinventing them: where data lives, how it is named, who can see it, how it moves between environments, and who pays for the compute.

Four planes, not one

When people draw a lakehouse they draw the data plane: sources on the left, bronze, silver and gold in the middle, dashboards on the right. That picture is correct and incomplete. An enterprise platform has four planes, and the three that are usually missing from the diagram are the ones that decide whether it survives contact with audit, security review and the finance team.

The ownership column is the point. The data plane is federated: domains own their pipelines and their gold models. The other three planes are centralized, run by a small platform team, and expressed as policy rather than as tickets. That split — federated data, centralized guardrails — is what lets the platform scale to many teams without either chaos or a bottleneck.

Workspaces and environments: one metastore, many bindings

The foundation decision is the Unity Catalog metastore. There is one per cloud region, at the account level, and every workspace in that region attaches to it. That single fact eliminates most of the historical sprawl: grants, lineage and audit no longer live inside individual workspaces, so a new workspace does not create a new island of permissions.

Workspaces then become what they should have been all along — compute and collaboration boundaries, not data boundaries. We typically deploy a dev, a staging and a production workspace per business unit or per platform, and we separate data by environment at the catalog level:

Two details make this work in practice. First, workspace–catalog binding: production catalogs are bound only to the production workspace, so the separation is enforced by the platform, not by people remembering not to do something. Second, production writes come from service principals, via jobs deployed from version control. Humans read production; pipelines write it.

Catalog design: domain, environment, layer

Inside the metastore we use a three-part convention: catalog = domain + environment, schema = layer or data product, table = entity at a stated grain. The result is a three-level name that tells you, at a glance, who owns the data, how trustworthy it is, and what one row means.

-- catalog       . schema  . table
claims_prod   . bronze  . raw_fnol_events      -- one row per landed event
claims_prod   . silver  . claim                -- one row per claim version (SCD2)
claims_prod   . gold    . fact_claim_payment   -- one row per payment transaction
claims_prod   . serving . claim_kpis_daily     -- one row per claim per day
reference_prod. codes   . loss_cause           -- shared, read-only

The data plane: contracts between layers

Medallion architecture is widely adopted and widely misunderstood. The layers are not folders. Each one is a promise the layer makes to the one above it, and the value of the pattern comes entirely from keeping those promises.

We implement the gates with Lakeflow Declarative Pipelines expectations where pipelines are declarative, and with explicit reconciliation jobs where they are not — row counts and control totals checked between silver and gold, and promotion blocked when they disagree. The detail that matters most is that consumers read from gold and platinum only. The moment a dashboard or an agent reads silver directly, the layer contracts stop meaning anything, because nobody can change silver without breaking a consumer they did not know they had.

For a worked example of these contracts in a regulated domain — nine schemas, a conformed star schema and the modeling rules that keep premium and loss numbers honest — see our P&C insurance lakehouse blueprint.

Identity and network: the plane nobody draws

Security reviews do not fail lakehouses on encryption. They fail them on identity sprawl and uncontrolled egress. Our baseline:

• Identity flows one way: your IdP is the source of truth, SCIM provisions users and groups into the Databricks account, and grants are made to account-level groups — never to individuals, never to workspace-local groups.
• Jobs run as service principals, one per pipeline or per domain, with OAuth machine-to-machine credentials. Personal access tokens are disabled in production workspaces.
• Private connectivity for the control plane and for storage, and serverless egress restricted to an allow-list, so a notebook cannot quietly send data somewhere it should not go.
• Sensitive data is classified with tags, and row filters and column masks are attached in Unity Catalog, so the same rules apply whether the reader is a dashboard, a notebook, a JDBC client or an AI agent.

Cost and operations: govern from the system tables

The finance conversation goes badly when the only answer to "what did this cost and who spent it?" is a bill. Unity Catalog's system tables change that: usage, billing, audit, lineage and query history are queryable tables, which means cost governance can be code.

-- Last 30 days of spend by team tag and product
SELECT  custom_tags['team']    AS team,
        billing_origin_product,
        SUM(usage_quantity)      AS dbus
FROM    system.billing.usage
WHERE   usage_date >= current_date() - INTERVAL 30 DAYS
GROUP BY 1, 2
ORDER BY dbus DESC;

Around that we put four controls: compute policies that cap instance types and enforce auto-termination; mandatory tags for team and cost center on every job and warehouse; budget policies and alerts per domain; and serverless as the default for SQL and jobs where the workload allows, which removes idle-cluster waste rather than monitoring it.

Six decisions that make or break the platform

These are the choices we make on every Databricks POD engagement. Each exists because we have seen the alternative cost a client months.

1. One metastore per region, decided on day one

Multiple metastores in one region re-create the islands Unity Catalog exists to remove. If a subsidiary genuinely needs isolation, use separate catalogs with separate bindings and grants, or Delta Sharing between metastores — not a second metastore by default.

2. Environment lives in the catalog, not the schema

Catalog-level separation lets you bind production to the production workspace and keep grants coarse and auditable. Schema-level separation forces fine-grained grants that drift.

3. Humans read production; pipelines write it

Every production write comes from a job running as a service principal, deployed from Git through Databricks Asset Bundles or your CI/CD tool. This single rule removes most production incidents and most audit findings.

4. No mounts, no keys on clusters

Storage credentials and external locations make file access governable. Mount points and keys in cluster configs make it invisible. There is no good reason to keep the latter in a new build.

5. Consumers never read silver

Publish gold and serving tables, and views on top of them, as the only supported interfaces. Silver is internal to the domain, and keeping it that way is what lets the domain fix it.

6. Metadata is a release gate

Table comments, column descriptions, owner and classification tags are checked in CI. It is the cheapest control in the platform, and the one that decides whether analysts — and language models — can find and trust your data later.

A checklist you can run against your own estate

The bottom line

A Databricks lakehouse becomes an enterprise platform when it stops being designed workspace by workspace. One metastore per region; catalogs by domain and environment, bound to the right workspaces; layers with real contracts and gates; identity flowing from your IdP into groups and service principals; and cost governed from tables rather than invoices.

None of it is exotic, and all of it is far cheaper to decide at the start than to retrofit. The organizations that get it right find that every new domain, every new dashboard and every new AI agent lands faster than the last. The ones that skip it find the opposite.