Service line 02 of 06
Data Warehouse & Lakehouse
Medallion lakehouses and cloud warehouses, designed new or migrated off legacy systems, with every migrated table reconciled before the old one is switched off.
When the warehouse becomes the bottleneck
Warehouse work usually starts with one of these.
The on-premises warehouse is close to capacity, and the licence renewal is coming up.
We size a cloud target, plan the move in waves by subject area, and run old and new side by side until the numbers match.
The data lake has become a pile of files nobody trusts.
We put the data under a table format with schemas and history (Iceberg or Delta) and rebuild it as bronze, silver and gold layers.
Business logic lives in thousands of lines of stored procedures.
We rewrite it as tested dbt models in version control, one subject area at a time, and compare every output with the old one.
Queries that took seconds now take minutes, and the bill grows with them.
We profile the heaviest queries and jobs, fix partitioning, clustering, file sizes and compute sizing, and put budgets and alerts in place.
Everyone can read everything, or getting access takes a chain of emails.
Access is granted per domain and role in Unity Catalog, Purview or the platform’s own controls, through groups in your identity provider.
One migration wave, planned in writing
Each wave moves one subject area. The plan names what moves, how the numbers are reconciled and who signs off before the old system is switched off.
An illustrative example, not client data.
What a migration moves, and how we check it
Nothing is switched off until the new platform gives the same answers as the old one. Every migration wave runs through this list.
Tables and their history
Land in bronze and silver tables in Iceberg or Delta. Row counts and column checksums are compared with the source.
Stored procedures and views
Become dbt models under version control. Their outputs are compared with the legacy results for the same period.
Scheduled jobs
Become orchestrated pipelines with retries and alerts. Both platforms run in parallel until their results agree.
Reports and extracts
Are repointed to gold tables. Each report moves once its owner has signed off the totals.
Users and permissions
Become roles in the platform’s catalogue, reviewed with each data owner before anyone gets access.
What the platform work covers
Each line ends in artefacts your team keeps, whoever runs the platform afterwards.
Architecture and migration
A target design that fits your cloud, your data volumes and your team, and a migration plan sequenced by subject area with a parallel run for each wave.
You receive
- Architecture decision records
- Migration plan with switch-off criteria
- Reconciliation report per wave
Bronze, silver and gold layers
Raw data kept as it arrived, cleaned and conformed in silver, and business-ready tables in gold, built as dbt or Spark models with tests.
You receive
- Modelled layers in your repository
- Catalogue entries with owners
- Lineage from source to gold
Cost and performance tuning
Query and job profiling, storage layout, compute sizing and scheduling, with spend tracked against a budget per workload.
You receive
- Tuning report with the changes made
- Cost dashboard and budget alerts
Governance and access
A catalogue, access policies per domain and role, and audit logging, set up in the platform’s native tools.
You receive
- Documented access model
- Audit logging switched on
Engines, table formats and catalogues
The platform choice comes out of the assessment. It depends on your cloud, your workloads and what your team already knows.
Listing a technology does not imply a vendor partnership or certification.
Lakehouse & warehouse
- Databricks
- Snowflake
- Microsoft Fabric
- BigQuery
- Redshift
Table formats
- Apache Iceberg
- Delta Lake
Transformation
- dbt
- Apache Spark
Governance
- Unity Catalog
- Microsoft Purview
Quality & monitoring
- Great Expectations
- Monte Carlo
Questions about warehouses and lakehouses
Not answered here? Ask in the request form.
Databricks, Snowflake, Fabric or BigQuery: which should we pick?
During the assessment we compare the realistic options against your own workloads and write down why we recommend one.
Do we need a lakehouse, or is a warehouse enough?
If your data is mostly structured and used for reporting, a cloud warehouse is often enough. A lakehouse pays off when you also keep files, events or training data for machine learning, or want tables more than one engine can read.
Can we migrate without stopping reporting?
Yes. Old and new run in parallel for each wave, and a report moves only after its owner has signed off the totals.
How do you keep costs from creeping back up?
Budgets and alerts per workload, a cost dashboard by team, and a tuning checklist in the runbook so your team can repeat what we did.
Will we be locked into the platform?
Data stays in your cloud storage in open formats, Iceberg or Delta, and transformations are dbt code. Changing engines later means rework, not a rebuild.
Discuss your platform
Bring a rough inventory: how many tables, jobs and reports you have, what they run on, and when the current licence or hardware runs out.
How it runs: a call (30 minutes with a senior engineer), an assessment and written plan, the build with weekly demos, then handover or run. Our delivery standard