Case study · Financial services · Cloud migration

On-premise SQL Server EDW to a Databricks Lakehouse on Azure for a leading financial firm

A leading financial firm moved its on-premise SQL Server enterprise data warehouse to a Databricks Lakehouse on Azure, with historical and incremental pipelines and the controls its regulators expect.

Industry
Financial services
Capability
Cloud migration, data platform
Platform
Databricks on Azure
Related POD
Databricks POD

Context

The client is a leading financial firm with a strong presence in investment banking, wealth management and retail banking. It relies on its data warehouse for risk management, fraud detection and regulatory reporting.

The problem

The on-premise SQL Server EDW was struggling to scale with growing data volumes and performance demands. The firm wanted a more scalable, efficient and cost-effective platform.

Three constraints shaped the migration: keep downtime and business disruption to a minimum, move large volumes of historical data through pipelines that process it efficiently, and meet strict regulatory requirements while keeping data secure.

What was built

VerticalServe designed a Databricks Lakehouse on Azure and the pipelines that fill it.

Flow: the on-premise SQL Server EDW is loaded through Azure Data Factory historical and incremental pipelines into a Databricks Lakehouse on Azure, with validation, encryption, access controls and monitoring. ON-PREMISE SQL Server EDW PIPELINES Azure Data Factory Historical load Incremental load AZURE Databricks Lakehouse (Delta) Advanced analytics Data validation · encryption · access controls · monitoring

Flow: the on-premise SQL Server EDW is loaded through Azure Data Factory historical and incremental pipelines into a Databricks Lakehouse on Azure, with validation, encryption, access controls and monitoring.

  • Lakehouse design: a Databricks Lakehouse on Azure designed around good practice for data storage, access and governance.
  • Historical and incremental pipelines: built with Azure Data Factory and Databricks Delta for ingestion, transformation and loading.
  • Data validation and testing: comprehensive checks to confirm data integrity and accuracy during the migration.
  • Security and compliance: data encryption, access controls and monitoring to meet regulatory requirements and protect sensitive data.

Stack

Databricks Lakehouse Azure Azure Data Factory Databricks Delta SQL Server (source)

How it was delivered

VerticalServe ran the migration in phases:

  1. Assessment: analysed the existing EDW architecture, data sources and requirements to set the migration strategy.
  2. Lakehouse design: designed the target platform on Azure.
  3. Pipeline development: built the historical and incremental pipelines.
  4. Validation and testing: verified integrity and accuracy of migrated data.
  5. Security and compliance: implemented encryption, access controls and monitoring.
  6. Knowledge transfer: training and documentation for the client’s IT team, who manage and support the platform.

The result

  • Better scalability and performance, with faster query execution and data analysis.
  • Lower infrastructure and maintenance costs through Azure’s pay-as-you-go pricing.
  • Stronger security and compliance with industry-specific regulations.
  • Advanced analytics capabilities with Databricks, for better insight and decisions.
  • Simpler data management and governance processes.

The original write-up reports these outcomes without figures.

What it means for a similar team

  • Plan two kinds of pipeline from the start: one for history, one for ongoing increments. That split is how both the history and the ongoing change were carried across here.
  • Validate data as it moves. In a regulated firm, proof that the new platform matches the old one is part of the deliverable.
  • Budget for knowledge transfer. The client’s own IT team runs this platform now.

This case study was first published on our Medium publication. Original article

Engage

Start a similar POD

Tell us the outcome you want and your constraints. We will reply with a discovery proposal.

Talk to us

Working on something similar?

An Innovation POD is a dedicated team that builds inside your environment. Tell us what you need to ship.

Start a POD