From Lakehouse to AI Agents: The Serving and Governance Layer

An agent is a new kind of user

For twenty years the lakehouse, and the warehouse before it, served two kinds of consumer: people looking at dashboards, and pipelines feeding other systems. AI agents are a third. They read data like an analyst, call systems like a pipeline, and act at a speed and volume neither ever did. And they make decisions about what to read and what to do next at run time, from instructions written in natural language.

That combination breaks a quiet assumption most data platforms rely on: that whoever queries a table already knows what they are allowed to do with it. An agent knows only what its tools and permissions allow. If the only thing between an agent and your claims data is a service account with SELECT on everything, the agent has exactly that access — for every user who talks to it.

The architecture

The serving and governance layer sits between the lakehouse and the agent. It has five parts: governed data for agents to read, tools that are themselves governed objects, a serving tier with a gateway, an identity model, and an audit layer that captures evidence from every hop.

Data for agents: gold, served three ways

Agents should read the same trusted data your dashboards do, which means gold and serving tables, never bronze or silver. What changes is how that data is shaped for them:

Two details matter more than they look. First, the vector index is built from a governed table, with the access-control attributes carried as metadata so retrieval can be filtered by who is asking. An index built from a folder of exported PDFs has no governance at all. Second, table and column comments are now part of the interface. Agents, like Genie, choose tools and columns from descriptions. A gold table without good comments is a gold table an agent will misuse.

Tools, not tables

The single most important design decision is that agents do not write SQL against tables. They call tools, and every tool is a governed object with an owner, a description, a signature and its own grants. On Databricks the natural form for read tools is a Unity Catalog function:

-- A read tool: narrow, typed, documented, governed like any other object
CREATE OR REPLACE FUNCTION claims_prod.agent_tools.open_claims_for_policy(
    policy_id STRING COMMENT 'Policy number, e.g. PA-2041-118'
)
RETURNS TABLE (claim_id STRING, loss_date DATE, status STRING, reserve DECIMAL(18,2))
COMMENT 'Open claims on one policy. Use when a user asks about the status of their claims.'
RETURN SELECT claim_id, loss_date, status, reserve
       FROM   claims_prod.gold.v_claim_current
       WHERE  policy_number = policy_id AND status <> 'CLOSED';

GRANT EXECUTE ON FUNCTION claims_prod.agent_tools.open_claims_for_policy
      TO `agent-claims-assistant`;

A narrow tool limits what the agent can ask for, makes its behavior testable, and gives you one place to change logic without retraining or re-prompting anything. It also composes with the catalog: the function reads a view whose row filters and masks still apply.

Action tools — anything that changes the business, such as settling a claim, issuing a refund or closing an incident — are different. They should call the system of record through its own API, and the authority limits must live in that system, not in the agent’s prompt. When we built Vsure, our carrier run by AI agents, the claims system refused any settlement above the adjuster agent’s limit and raised an approval to a supervisor. The agent could not talk its way past it, because the limit was never the agent’s to enforce.

Serving and the gateway

Agents and the models they call run behind Mosaic AI Model Serving, with the AI gateway in front. The gateway is where the platform-wide controls live, so individual teams do not each reimplement them:

• Rate limits and usage tracking per endpoint, per user and per agent, so one runaway loop cannot consume the month’s budget.
• Guardrails on inputs and outputs: PII detection, unsafe content filters and topic restrictions, applied before anything reaches the model or the user.
• Payload logging to inference tables in Unity Catalog, so every request and response is queryable and governed like any other data.
• One front door for every model, hosted or external, so switching providers is a configuration change and not a security review.

Agent code, prompts and tool definitions are versioned together in MLflow, traced with MLflow Tracing, and released only after they pass an evaluation set — automated judges plus human review on a sample — with no regression against the previous version.

Whose identity does the agent use?

This is the question most pilots skip, and the one security teams ask first. There are two valid answers and one invalid one:

The on-behalf-of model is the safest default for assistants because it makes the agent unable to show a user anything the user could not already query. The service-principal model is right for autonomous work, provided each agent has its own principal — so its access can be reviewed, its activity audited and its credentials revoked without affecting anything else.

Three trust boundaries

It helps to name where risk actually crosses. There are three boundaries, and each needs a different kind of control:

Audit and assurance: evidence of what changed

Tracing tells you what an agent did: prompts, tool calls, tokens, latency. That is necessary and not sufficient. When an agent is allowed to change something that costs money, you also need to know what changed in the business, under whose authority, at what cost — and you need to be told when a control quietly stops firing.

This is the gap our InsightLense product is built for. It organizes agent evidence into five layers:

Because traces, inference tables and audit logs all land in Unity Catalog, this evidence can be joined to the business data itself. That is what turns a log into an answer an auditor accepts.

The control checklist before an agent touches production

The bottom line

Agents do not need a new data platform. They need the lakehouse you should already have — governed gold data, permissions in the catalog, good metadata — plus a thin serving and governance layer that makes the agent a well-behaved citizen of it: tools instead of tables, its own identity or the user’s, a gateway in front, limits in the systems it acts on, and evidence from every step.

Build that layer once and every new agent inherits it. Skip it and every agent becomes its own security review, which is why so many pilots never reach production.