Streaming · Streamhouse · AI Agents

Introducing the Streamhouse: The Lakehouse Pattern, Moved to the Current State of the Business

At Current 2026, Confluent introduced the Streamhouse: an architecture that continuously captures, processes, governs and serves the current state of a business to applications, analytics and AI agents. Our Kafka POD has spent years building that pattern piece by piece for clients, with Kafka, Flink, CDC and streaming into Iceberg and Delta. This post covers what was announced, how it relates to the lakehouse, the use cases where it pays off, what it improves over the patterns we built before, and how we help teams adopt it.

TL;DR

A lakehouse answers “what happened?”. A Streamhouse answers “what is true right now?”, with the same governance, and serves the answer to applications, dashboards and AI agents. Confluent’s version adds Query as a fifth pillar next to Stream, Connect, Process and Govern. Query has three ways in: Tableflow (streams as open tables for the lakehouse), Lightning Tables (low-latency SQL and API access to current state, now generally available on AWS) and the Real-Time Context Engine (current context for agents over MCP). It complements the lakehouse rather than replacing it. Our Kafka POD has been building this pattern by hand for years, with Flink-derived views, CDC, and Iceberg and Delta sinks feeding a separate serving database or cache. The Streamhouse removes much of that custom plumbing. Below: the architecture, the use cases, what it improves, which lakehouse benefits carry over, the caveats, and how we help.

What was announced

On November 4, 2026, at Current 2026 in San Francisco, Confluent (now an IBM company) introduced its Streamhouse architecture. Confluent defines a Streamhouse as a data architecture that lets AI agents and applications access and act on a governed view of the current state of the business, at machine speed, and run at production service levels.

The announcement has four parts:

CapabilityWhat it does
Query, the fifth pillarJoins Stream (Kafka), Connect (managed and custom connectors), Process (Apache Flink) and Govern (schemas, lineage, catalog, quality, access control). It serves the current state of the business to whoever needs it, through the interface that fits.
Lightning TablesA read-only serving layer that materialises Kafka data into durable tables and keeps them current. Applications query it with SQL or an API, and agents through MCP. In Confluent’s internal testing: about 100 ms from event to queryable, 10–100 ms query latency and 1,000+ queries per second. Generally available on Confluent Cloud on AWS.
TableflowExposes topics as open tables (Apache Iceberg) for lakehouses and warehouses, so analytics reads the same governed data without a separate ingestion job.
Real-Time Context EngineServes current business context to AI agents over the Model Context Protocol, with OAuth, customer-managed keys and usage metrics.

Confluent also announced a schema compatibility policy that catches breaking changes before they reach Flink, Tableflow or Lightning Tables. It also announced IBM Granite time-series models running inside Flink for forecasting and anomaly detection, and integration with watsonx.data, which queries Tableflow’s Iceberg tables without copying them.

A pattern, not just a product. In September, Aiven, Confluent, Redpanda, StreamNative and Ververica formed a Streamhouse Working Group and published a shared definition. It allows open-source, commercial or mixed implementations. That matters to us because we deliver on Confluent, Redpanda, Apache Kafka and MSK, and the architecture is the same across all of them.

The “what is true right now” gap

Confluent’s launch example is an airline. A flight is cancelled, and a rebooking agent has to offer the passenger a new connection. Flight operations knows where the aircraft is, scheduling knows the new departure time, and reservations knows which seats are free. Each system updates on its own clock. When the agent reads them at different moments, it can book a connection the aircraft cannot make.

We see the same gap in every industry we work in. An insurance claims agent needs the policy as endorsed this morning, not as of last night’s load. A fraud check needs the card’s last five swipes, not yesterday’s aggregate. A rental pricing engine needs current fleet availability. The lakehouse does a good job of answering questions about history. It was never designed to serve the current state of an entity in milliseconds to thousands of callers.

So teams built a second path, which usually looks like this:

  • Kafka carries the events.
  • A Flink or Kafka Streams job builds the current-state view.
  • That view is written to a side database or cache (Redis, Postgres, DynamoDB, Cassandra) so applications can query it.
  • A separate sink lands the same events in the lakehouse for analytics.
  • Each AI use case then builds its own extract for retrieval or agent context.

Every arrow in that list is a pipeline someone has to own, secure, monitor and keep in sync with schema changes. The Streamhouse is an attempt to make that second path a first-class part of the streaming platform.

The Streamhouse reference architecture

Streamhouse: one governed stream, three ways to read it
Sources Stream Process Query Consumers Operational DBsCDC (Debezium) Applicationsdomain events Mainframe & ERPconnectors SaaS & devicesmanaged connectors Kafka Confluent · Redpanda Apache · MSK events in motion retention for replay SchemaRegistry · contracts Apache Flink join · enrich · dedupe customer 360 order / claim state ML features derived once, reused Tableflow topics as Iceberg tables history · time travel Lightning Tables current state · SQL / API ~100 ms fresh · read-only Real-Time Context Engine · MCP OAuth · usage metrics LakehouseDatabricksSnowflake Appsmicroservicesdashboards AI agentscopilotsMCP clients Govern: schemas · lineage · catalog · data quality · encryption · access policyapplied once at capture and enforced through processing and every read path Still owned by operational systems: transactions, writes, reservationsthe Streamhouse serves state; it does not replace the system that commits a booking or a payment
The stream carries events. Flink derives business entities from them once. The Query layer serves each consumer through the interface it needs: Iceberg for history, SQL or an API for current state, MCP for agents. Governance is set once and applies to every read path.

Streamhouse and lakehouse: complements, not competitors

Confluent positions the Streamhouse as an operational complement to the warehouse and lakehouse, and that matches what we see in client estates. The lakehouse is still the right home for history, exploration and model training. The Streamhouse is the right home for the current state of each entity, served fast and consistently.

LakehouseStreamhouse
Question it answersWhat happened, and why?What is true right now?
FreshnessMinutes to hours (minutes with streaming ingestion)Sub-second to seconds
Typical readerAnalysts, BI, data science, model trainingApplications, operational dashboards, AI agents
Access patternLarge scans, joins, aggregatesKey lookups and small filtered queries at high concurrency
Storage of recordOpen tables (Iceberg, Delta) on object storageThe event log, plus materialised current-state tables
Shared foundationOpen formats, one schema contract, one governance model and one lineage graph. With Tableflow, the stream writes the lakehouse table directly.
How the pattern evolved in our client work
1 · Batch lakehouse Nightly ETL → lake freshness: hours stream and lake are two copies that drift apart “whose number is right?” 2 · Streaming lakehouse Kafka + Flink → Iceberg / Delta freshness: minutes one governed table per entity current state still lives in a hand-built cache or side database extra system to sync and secure 3 · Streamhouse One stream, three read paths history: open tables current state: SQL / API agents: MCP context governance applied once fewer systems, one truth
Most of our streaming clients today sit in stage 2. The Streamhouse is a productised stage 3, and the step from 2 to 3 is much smaller than the step from 1 to 2.

How we have been building this pattern for clients

The Streamhouse gives a name and a managed product to an architecture our Kafka POD has been assembling from open components for years. A few examples from our case-study library:

1

Real-time point-of-sale fraud detection for a credit card company

Card transactions streamed through Kafka, with stateful processing that keeps a current profile per card and scores each swipe against it. That per-card current state is exactly what a Lightning Table now serves directly. We later extended the pattern to an AI fraud-investigation agent on Redpanda.

2

Real-time rental pricing on Confluent Kafka

Fleet availability, demand and competitor signals joined in the stream to keep a current price per location and vehicle class. Pricing services read the current state, and the full history landed in the lake for model training: the two read paths a Streamhouse formalises.

3

Real-time call insights for a contact center, and viewership analytics on Confluent Cloud

Streaming events enriched with AI/ML inference while the call or session is still live, with the results served to supervisors and agents in the moment and kept for trend analysis afterwards.

4

Streaming into Iceberg and Delta, and Kafka as the context layer for agents

We have written up the mechanics in Streaming Into the Lakehouse (exactly-once, CDC upserts, table maintenance) and in Kafka as the context layer for AI agents. Together with mainframe offload to Kafka and our MSK-to-Confluent Cloud migrations, these are the foundations a Streamhouse sits on.

In each engagement the hardest parts were the same: agreeing on schemas and ownership, keeping the side database in sync with the stream, and proving to auditors who could see what. Our accelerators exist to shorten that work.

Kafka Designer: one of our Kafka POD accelerators
Kafka Designer screenshots: a cluster inventory across Confluent Cloud, Confluent Platform and Apache Kafka with migrate, clone and template actions; a visual cluster designer showing brokers, connectors and ZooKeeper nodes; connector configuration; generated Python producer code; and Prometheus/Grafana monitoring templates.
Kafka Designer gives one view of clusters across Confluent Cloud, Confluent Platform and Apache Kafka. It manages connectors, generates client code and monitoring templates, and drives migrations. In a Streamhouse rollout it provides the inventory and the code-as-config baseline we start from.

What the Streamhouse makes better

Compared with the hand-built stage-2 pattern, here is where we expect the biggest improvements for clients:

  1. Fewer systems to run. For suitable workloads, Lightning Tables replaces the Redis, Postgres or DynamoDB side store that teams build only to make stream state queryable. That means one less pipeline, one less security review and one less sync job to debug.
  2. Derive once, reuse everywhere. A customer 360, claim state or order state is built once in Flink and read by apps, dashboards, the lakehouse and agents, instead of each team rebuilding it from raw topics.
  3. History and current state agree. Tableflow writes the lakehouse table from the same stream that feeds the current-state table, so the operational and analytical views stop drifting apart.
  4. Agent-ready by default. MCP access through the Real-Time Context Engine gives agents governed, current context without a bespoke retrieval service per use case.
  5. Governance follows the data. Schemas, lineage and access policy are applied at capture and enforced through every read path, and the new compatibility policy blocks breaking changes before they reach consumers. For agents that act on data, this matters more than it ever did for dashboards.
  6. A built-in feature store. Features computed in Flink are stored for ML serving, which narrows the gap between training data in the lakehouse and inference data in production.
  7. Reach into legacy systems. With IBM, the platform reaches further into IBM Z mainframes and enterprise databases, the sources that are often the real system of record in insurance and banking.

Use cases where it pays off first

IndustryCurrent-state questionWho reads it
InsuranceWhat is this policy’s coverage, endorsements and open claims right now? Is this FNOL a duplicate?Claims agents, broker portals, underwriting copilots
Banking & paymentsWhat are this card’s last transactions, risk score and dispute status?Fraud engine, investigation agents, customer service
Retail & CPGWhat stock is available at this store, and where is this order?Order-change agents, store apps, replenishment
Rental & mobilityWhich vehicles are available here, and what is the current price?Pricing services, booking apps, fleet operations
Travel & logisticsWhere is the aircraft or shipment, and can this connection still be made?Rebooking agents, customer notifications
Contact centersWhat has this customer done in the last ten minutes across channels?Live agent assist, supervisors
Manufacturing & utilitiesWhat is this asset’s current state, and is it drifting from normal?Operations dashboards, anomaly detection (for example, Granite time-series models in Flink)

The best first use case is a narrow current-state lookup that today depends on a side database: order status, policy status or card profile. It is easy to measure (latency, freshness, one system retired) and easy to expand once it works.

The lakehouse benefits that carry over

Clients adopted the lakehouse for a consistent set of reasons. Each one has a direct counterpart in the Streamhouse:

What companies get from the lakehouseHow it applies to the Streamhouse
One copy of the data instead of a lake plus several warehouse extractsOne stream feeds history (Tableflow), current state (Lightning Tables) and agents (MCP), so there are no side copies to reconcile
Open formats and no lock-in to one query engineIceberg tables from the stream are readable by Databricks, Snowflake, Trino, Spark and watsonx.data. The Streamhouse definition is shared across five vendors.
Unified governance: one catalog, lineage and access modelThe schema contract, lineage and access policy are set at capture and enforced on every read path, including agent access
Lower cost from cheap object storage and retired ETLFewer ETL jobs, caches and serving databases. History stays on object storage, and only current state is materialised for fast reads.
BI and ML on the same platformAnalytics, ML features and operational apps share one derived view. Training data and inference data come from the same pipeline.
Faster time to insightNew consumers subscribe to existing governed views instead of commissioning a pipeline. At launch, Accenture described time-to-insight dropping “from months to days” on its Streamhouse.
Reliable, ACID tablesExactly-once processing and durable, schema-enforced tables, with the event log kept for replay and audit

What to plan for

We are enthusiastic about the pattern, but we also want clients to go in clear-eyed. Independent analysts have raised the same points we would:

1. Read-only by design. Lightning Tables serve state. They do not enforce transactions. A reservation or payment still needs its own check in the operational system before an agent acts.

2. Current is not the same as consistent. Teams still have to decide which source is authoritative for each field, how late events are handled, and when incomplete data should block an action rather than allow it.

3. Deployment coverage. Lightning Tables is generally available on Confluent Cloud on AWS today. Self-managed, Azure and GCP estates need a bridge plan: a Flink-maintained serving store now, then migration when coverage arrives.

4. Vendor latency figures are not end to end. The 100 ms and 10–100 ms numbers come from internal testing of the serving layer. Measure source-to-action latency in your own environment.

5. Agents need an audit trail. Current state cannot reconstruct a past decision. Log what context an agent read and when, and map source-system permissions to agent access explicitly.

6. Continuous processing has a running cost. Always-on Flink, storage and query capacity need a cost model per use case, and a clear owner between the data platform and AI budgets.

How VerticalServe helps you adopt the Streamhouse

Our Kafka POD is 3–6 senior streaming engineers working inside your environment. For Streamhouse adoption we offer:

  1. Streamhouse readiness assessment (2–3 weeks). We inventory your topics, side databases, caches and lakehouse ingestion jobs, identify which current-state lookups are candidates, and produce a sequenced roadmap with a cost model.
  2. Schema and data-contract foundation. Schema Registry compatibility policy, topic ownership and naming, and PII classification, so every read path inherits governance. Our Schema UI and Stream DLP & Catalog accelerators speed this up.
  3. Flink-derived business entities. We design and build the customer, policy, order or card views once, with correct windowing, state sizing and reprocessing strategy.
  4. Tableflow and open-table integration. Streams landed as Iceberg (or Delta through our Databricks POD) and wired into Databricks, Snowflake, Trino or watsonx.data, with catalog and lineage in place.
  5. Lightning Tables pilot. We pick one side database to retire, run the old and new paths in parallel, reconcile them, and cut over only on evidence.
  6. Agent context over MCP. We connect the Real-Time Context Engine to your agents (Gemini Enterprise, Claude, Copilot or custom), with OAuth scopes, decision logging and guardrails for stale or partial data.
  7. Vendor-neutral delivery. On Redpanda, Apache Kafka or MSK, or where Lightning Tables is not yet available, we build the same architecture with Flink and an open serving layer, so the design carries over when you are ready to move.
  8. Migration to Confluent Cloud. Our automated migration tooling moves producers, consumers, Streams apps and Flink jobs from self-managed Kafka or MSK, run in parallel with reconciliation before cutover.
  9. Operate and enable. Lag and freshness SLOs, DR rehearsal, cost tuning, runbooks and paired delivery until your team runs it.

Once current state is governed and queryable, our AI products can read it. InsightCompose, for example, turns governed data into verified insights, dashboards and data APIs, each published as a versioned endpoint in one catalog. Pointed at Streamhouse tables, those insights describe the business as it is now, not as of last night.

InsightCompose: governed insights and data APIs on top of current data
InsightCompose home page: an AI insights platform where an agent reads your data, proposes questions, checks every figure against source rows, and publishes insights, dashboards and data APIs as versioned endpoints in one governed catalog.
InsightCompose runs inside your environment with your own LLM and checks every figure it reports against the rows it came from. With a Streamhouse underneath, the insights are current as well as verified.

A practical first 90 days

1

Weeks 1–3: Assess

Map current-state lookups and the side stores behind them. Choose one use case with a measurable win. Agree on freshness and latency targets with the people who read the data.

2

Weeks 4–6: Foundation

Set schema contracts and compatibility policy. Build the Flink-derived entity. Land it with Tableflow so analytics sees the same data from day one.

3

Weeks 7–10: Serve

Stand up a Lightning Table (or the open equivalent). Run it in parallel with the existing side store, reconcile, and measure end-to-end latency.

4

Weeks 11–13: Act

Cut the application over, retire the old store, and connect the first agent through MCP with decision logging. Hand over runbooks and SLOs.

Streamhouse readiness checklist

AreaReady when
Use caseOne current-state lookup chosen, with a freshness target and a side store to retire
SourcesThe system of record is streaming (CDC or events), with topics keyed by entity ID
ContractsSchemas registered, with compatibility rules that block breaking changes
Derived viewsEach entity built once in Flink, with an owner and a documented definition
AuthorityThe authoritative source per field and the rules for late events are written down
Read pathsEach consumer mapped to Tableflow, Lightning Tables or MCP
AccessSource permissions mapped to app and agent scopes; agent reads logged
WritesTransactional actions still checked in the operational system
ObservabilityLag, freshness and query-latency SLOs alerting end to end
CostProcessing, storage and query cost modelled per use case, with an owner

The bottom line

The lakehouse gave enterprises one governed copy of their history. The Streamhouse applies the same idea to the present: one governed stream that serves history to analytics, current state to applications and context to AI agents, without a fleet of side databases in between.

For teams already running Kafka and Flink, it is an evolution rather than a rebuild. Most of the hard work is the same work we have done with clients for years: schema contracts, Flink-derived entities, clean streaming into open tables, and governance that holds up to an auditor. Start with one current-state lookup, prove it, retire a system, and then let agents read from it.

Product details, availability and performance figures are as announced by Confluent on November 4, 2026 and reported in its press release and independent analyst coverage. Verify current availability for your cloud and region.

Planning your Streamhouse?

Our Kafka POD runs a 2–3 week Streamhouse readiness assessment: current-state candidates, side stores to retire, schema and governance gaps, and a sequenced plan. We deliver on Confluent, Redpanda, Apache Kafka or MSK, inside your environment.

Kafka POD
Back to blog