← All articles Use case

Use case: moving a data platform without losing track of what depends on what

A made-up retailer's data team migrates warehouses and Kafka clusters with Insight Catalog for lineage and impact analysis, Insight Automation Hub for validated migration runs, InsightKafka for cluster health, and Insight Dataflow for the new pipelines.

Example · a made-up retailer

Harbourline Home moves off its on-prem warehouse

Harbourline Home (made up) runs 200 stores and an online shop. Its data platform is an on-prem warehouse fed by nightly jobs and two self-managed Kafka clusters for point-of-sale and inventory events. The plan is a managed cloud warehouse and a managed Kafka service within a year. The risk isn't moving the data; it's breaking a dashboard or a forecast nobody knew depended on a table. Everything here is fictional.

The four steps

1. Map what depends on what: Insight Catalog

Before anything moves, the team connects Insight Catalog to the warehouse, the ETL jobs and the BI tool. It crawls and profiles schemas, tracks lineage through pipelines to dashboards and reports, and classifies sensitive data. The feature that matters most here is impact analysis: for any table or column, the downstream assets that would be affected by changing it. That turns "what will break?" into a list with owners.

2. Know the state of the clusters: InsightKafka

The Kafka clusters get an InsightKafka health check first: an automated assessment of each cluster with findings to fix or carry over deliberately, so the team doesn't faithfully migrate a misconfiguration.

3. Migrate with validation: Insight Automation Hub

Insight Automation Hub plans and runs migrations (databases, warehouses, data-processing workloads, Kafka clusters and BI) as workflows. Its distinguishing part is validation: test cases generated from the schema and data, row-level diffs, statistical drift checks and golden-query comparisons. Every workflow has rollback steps defined next to the forward steps; if validation fails, the run halts, alerts the team and rolls back to the last known good state. Each run records who launched it, which script version ran, every check, and every approval or override.

4. Rebuild the pipelines: Insight Dataflow

The nightly jobs are rebuilt in Insight Dataflow: a visual designer with connectors for the old and new platforms, generating Airflow DAGs that run the same on AWS, Azure and GCP, with lineage and monitoring. Catalog then picks up the new pipelines, so the lineage map stays true after the move.

How Harbourline sequences it

WaveWhat movesGate to the next wave
1Reference data and finance marts (few dependents)All golden queries match; no rollback
2Inventory event stream and its consumersKafka findings closed; consumer lag within agreed bounds
3Sales history and forecasting inputsForecast outputs compared side by side for two weeks
4Decommission the old platformCatalog shows no remaining dependents

Validation is only as good as its checks. Generated test cases and golden queries need review by the people who own the data. Agree what "matching" means for each dataset (exact, within tolerance, or statistically) before the first run.

What the audit trail is for

A migration of this size gets reviewed: by finance, because the warehouse holds the numbers; by security, because data is moving; and by the business, because forecasts depend on it. Automation Hub's run history answers the reviewers' questions directly: which script version ran against which tables, what every row count, checksum and golden-query comparison showed, and who approved or overrode a step. Catalog's lineage shows that each dependent asset was accounted for. Together they replace the spreadsheet of migration checks that usually goes stale halfway through.

Where it runs

Automation Hub can be used as cloud SaaS for a quick start, or self-hosted on Kubernetes or Docker in your cloud or data centre, with its control plane and workers running alongside your data, which is the usual choice when the data is sensitive.

Related: the reference architecture.

See the right Insight products in your environment

Tell us what you're trying to build, and we'll walk you through the products that fit, with your data, where it lives.

Request a demo

Keep reading

Guide
Ask + RAG + LenseCatalog + PromptsDLM

Guide: a phased adoption plan for the Insight platform

Where to start and what to add next: a first phase with Ask, RAG and Lense, a second with Catalog and Prompts, and a third with domain models and domain solutions, with the questions that decide when to move on.

November 17, 2026 · 8 min read
Use case
Insight AskInsight RAGInsight Lense

Use case: one search box for IT operations, across tickets, wikis and code

How a made-up manufacturer's IT operations team uses Insight Ask on top of Insight RAG: connect ServiceNow, Confluence, Jira, Slack and Git, keep permissions intact, and trace every answer with Insight Lense.

November 13, 2026 · 7 min read
Use case
Insight UWInsight ExtractInsight Lense

Use case: commercial insurance submissions, from broker email to underwriter decision

How a made-up specialty insurer combines Insight Extract and Insight UW: broker emails and attachments become validated data, each submission is assessed against guidelines and appetite, and the underwriter decides with the evidence in front of them.

November 11, 2026 · 8 min read