← All articles How it works

A reference architecture: how Ask, RAG, Lense, Prompts and Catalog fit together

The layers of a production enterprise AI system and which Insight product covers each: data and metadata, ingestion and retrieval, the applications people use, and the governance plane around all of them.

A demo needs a model and a prompt. A production system needs a lot more: connectors to where knowledge actually lives, pipelines that keep it fresh, a retrieval layer that respects who's allowed to see what, applications people use, and a governance plane that can say afterwards what happened and why. This post maps those layers to the Insight platform products.

Sources and pipelines

Knowledge lives in collaboration tools (SharePoint, Confluence, Slack), work trackers (Jira, ServiceNow), code (Git), databases, warehouses and object stores. Insight Dataflow moves and prepares data between any of them: a visual designer with 100+ connectors, generating Airflow DAGs and running on engines you already operate. Documents that need fields pulled out reliably, such as forms, statements or submissions, go through Insight Extract, which runs multi-step agentic pipelines with per-field confidence and human review for exceptions.

Insight Catalog sits beside the pipelines: it discovers and profiles data assets, tracks lineage from source to dashboard, and classifies sensitive data such as PII. That classification is what lets the layers above enforce policy.

Retrieval

Insight RAG is the retrieval layer. It brings sources in through connectors, extracts text, tables and media with format-specific extractors, chunks and embeds content into a pluggable vector database, and captures business, security and classification metadata at ingestion, so access control is enforced at query time and every retrieval is audited. Redaction and tokenisation keep PII out of the index, and guardrails protect the model tier against prompt injection.

Models are your choice: hosted providers, private endpoints, or domain-specific small models built with Insight DLM, which curates enterprise data, generates training sets, and fine-tunes and evaluates open models.

Applications

On top sit the things people use. Insight Ask is search and answers across all connected sources, with entity relationships, summaries and browser extensions. Insight AI Assist is an assistant with specialised agents. And your own applications reach the same retrieval layer through APIs and SDKs.

The governance plane

  • Insight Prompts is the registry: prompts are versioned, tested against datasets with LLM-judge scoring, approved through role-aware workflows, and pulled by applications through an SDK.
  • Insight Lense traces every run: the steps an agent took, the authority it acted under (what it asked to do, what it was allowed, who it had to ask), and the business record that changed, with cost and scores on every step.
  • Insight Security Operations monitors data access, enforces tokenisation policies and watches for leaks.

The products are linked where it matters: Lense records which prompt version produced each trace, so you can check whether the version you promoted in Prompts actually did better.

Example · a made-up manufacturer

One question, through every layer

An engineer at a made-up equipment manufacturer asks Ask: "What torque spec did we change on the Series 4 pump last year, and why?" Ask queries RAG, which retrieves from the Confluence change log and a Jira ticket, filtering out a restricted supplier contract the engineer can't see. The prompt that frames the answer comes from the Prompts registry, version 7. Lense records the retrieval, the model call, the cost and the prompt version under one trace. If the answer is wrong, the team can see exactly which document it came from.

Start small. Nobody deploys every layer on day one. The suite's own suggestion is to start with Ask, RAG and Lense, then add Catalog, Prompts and DLM as the programme matures. See a phased adoption plan.

How the layers talk to each other

  • Metadata flows up. Classifications and ownership from Catalog, and permissions from each source, are attached at ingestion in RAG, so the application layer never has to re-implement access rules.
  • Prompts flow down. Applications fetch approved prompt versions from the Prompts registry instead of hard-coding them, so a change is reviewed once and applied everywhere.
  • Evidence flows sideways. Every layer emits traces to Lense over standard OpenTelemetry, so one query can follow a request from the search box to the documents and the model call.

Because the layers are separate products, you can also swap parts: bring your own vector database, LLM or catalog where you already have one, as the RAG site puts it.

See the right Insight products in your environment

Tell us what you're trying to build, and we'll walk you through the products that fit, with your data, where it lives.

Request a demo

Keep reading

Guide
Ask + RAG + LenseCatalog + PromptsDLM

Guide: a phased adoption plan for the Insight platform

Where to start and what to add next: a first phase with Ask, RAG and Lense, a second with Catalog and Prompts, and a third with domain models and domain solutions, with the questions that decide when to move on.

November 17, 2026 · 8 min read
Use case
Insight CatalogAutomation HubInsightKafkaInsight Dataflow

Use case: moving a data platform without losing track of what depends on what

A made-up retailer's data team migrates warehouses and Kafka clusters with Insight Catalog for lineage and impact analysis, Insight Automation Hub for validated migration runs, InsightKafka for cluster health, and Insight Dataflow for the new pipelines.

November 15, 2026 · 8 min read
Use case
Insight AskInsight RAGInsight Lense

Use case: one search box for IT operations, across tickets, wikis and code

How a made-up manufacturer's IT operations team uses Insight Ask on top of Insight RAG: connect ServiceNow, Confluence, Jira, Slack and Git, keep permissions intact, and trace every answer with Insight Lense.

November 13, 2026 · 7 min read