Blueprint 05 · Data + AI
AI Over Enterprise Data: Grounded Analytics & the RAG Flywheel
Let anyone ask a question in plain language and get a trustworthy answer over real organisational data. A tiered cascade resolves most questions before the model is ever called, the prompt is grounded in live schema, the output is validated and self-corrected — and every success makes the next answer better.
Self-service analytics without the foot-guns
"Ask your data anything" is an irresistible promise and a minefield. Point a language model at a database naively and you get invented columns, queries that scan whole warehouses, leaked rows the asker shouldn't see, latency measured in seconds, and a bill that scales with curiosity. This blueprint is the set of disciplines that turn the demo into a system you can put in front of real users.
The organising principle is that the LLM is the last resort, not the front door. A cascade of cheaper, deterministic resolvers sits in front of it; when a question does reach the model, it is heavily constrained — grounded in the actual schema, restricted to read-only output, validated against the data, and corrected in a loop. Success is captured so the system needs the model less over time.
Cost, latency and accuracy are usually treated as separate concerns. The cascade solves all three at once: every question answered by an exact match, a template, a cube or a cached-similar question is a question answered in milliseconds, for free, and deterministically. The model is reserved for the genuinely novel — where its cost and variance are justified.
Resolve cheap, ground hard, learn always
Tiered resolution cascade
Order resolvers cheapest-first: exact cached answer → pre-written query patterns → parameterised templates → pre-aggregated cubes → semantic (vector) cache → and only then the model. Most traffic never reaches the bottom.
Schema-grounded prompts
When the model is invoked, the prompt is built from live introspection — tables, columns, types, cardinalities, sample rows, curated descriptions — so it works from what exists and cannot invent structure.
Generate → execute → validate → correct
Generated queries are restricted to read-only, screened against an allowlist, executed, and checked for shape and intent. On error or an obviously wrong result, the error is fed back for a bounded number of self-correction attempts.
The RAG flywheel
Every validated question→query pair is embedded and stored with rich metadata. Future similar questions retrieve it as a worked example, so accuracy compounds with use — and human-validated entries are weighted highest.
Permission as source of truth
Access scope is taken from the session and injected into both the prompt and the executed query. A mature analytics/permission engine — not the AI layer — owns what data a user may see.
Recommend the view, not just the rows
A rule-based recommender maps the shape of a result to an appropriate visualisation, and the model produces a short narrative — turning a table into an answer a non-analyst can act on.
The AI layer is an orchestration tier. Query execution, charting and row-level permissions are best delegated to an established analytics/BI engine. Reusing that engine's access controls as the source of truth means the AI can never become a path around them.
The cascade, drawn out
flowchart TB
Q["Natural-language question
(typed or spoken)"] --> AUTH["Resolve identity + access scope
(from session)"]
AUTH --> T1{"Exact cached
answer?"}
T1 -->|hit| OUT
T1 -->|miss| T2{"Pre-written
query pattern?"}
T2 -->|hit| EXEC
T2 -->|miss| T3{"Parameterised
template?"}
T3 -->|hit| EXEC
T3 -->|miss| T4{"Pre-aggregated
cube covers it?"}
T4 -->|hit| EXEC
T4 -->|miss| T5{"Semantic cache
(vector) similar?"}
T5 -->|hit| EXEC
T5 -->|miss| LLM["Schema-grounded LLM
+ retrieved examples (RAG)"]
LLM --> GUARD["Read-only allowlist
+ identifier checks"]
GUARD --> EXEC["Execute query
(scoped by permissions)"]
EXEC --> VAL{"Valid &
sensible result?"}
VAL -->|no, retry n×| LLM
VAL -->|yes| OUT["Narrative + recommended chart"]
OUT --> FLY[("Knowledge base
embed validated Q→query")]
FLY -.retrieved by.-> LLM
The resolution tiers
| Tier | Resolves | Cost / latency |
|---|---|---|
| Exact match | Verbatim repeat of a known question | Near-zero, deterministic |
| Query patterns | Common questions with hand-written queries | Near-zero, deterministic |
| Parameterised templates | Known skeletons with extracted parameters substituted in | Near-zero, deterministic |
| Pre-aggregated cubes | Questions covered by a star-schema cube of dimensions/metrics | Low, deterministic |
| Semantic cache | Questions similar (by embedding) to a past validated answer | Low; one vector lookup |
| Grounded LLM | Genuinely novel questions | Highest; reserved for the long tail |
When the model is finally needed
sequenceDiagram
participant U as User
participant O as Orchestrator
participant K as Knowledge base
participant L as LLM
participant D as Data engine
U->>O: Question (scope from session)
O->>O: Cascade miss → go to model
O->>D: Introspect live schema
O->>K: Retrieve similar validated examples
O->>L: Grounded prompt (schema + examples + scope)
L-->>O: Candidate query (JSON)
O->>O: Read-only + identifier allowlist check
O->>D: Execute (permission-scoped)
alt Error or wrong shape
D-->>O: Error / empty
O->>L: Feed back error (retry, bounded)
else Valid
D-->>O: Rows
O->>U: Narrative + chart
O->>K: Embed & store validated Q→query
end
A first-pass generated query is often nearly right. Executing it and feeding the actual error — a missing column, a type mismatch, an empty result over a non-empty table — back to the model lets it fix its own mistake against ground truth. A small, bounded loop captures most of the accuracy gain without unbounded cost.
Five alternate use cases
Conversational sales analytics
Store managers ask "how did category X do last week vs last year?" Templates and cubes serve the routine questions instantly; the model handles the unusual ones, all scoped to the manager's region.
Self-service product metrics
Non-technical CSMs query usage data in plain language; the recommender turns results into charts they can paste into a QBR, with permissions ensuring each sees only their accounts.
Grounded policy & document Q&A
The same RAG spine answers questions over a corpus of documents instead of tables — retrieve passages, ground the answer, cite sources, and refuse when there is no match.
Ad-hoc reporting copilot
Analysts get a first-draft query they can inspect and edit in a SQL view; read-only enforcement and validation make it safe to expose against production replicas.
Operational dashboards on demand
Administrators ask about throughput, wait times and utilisation; sensitivity tiers and masked denials ensure clinical detail never surfaces to someone without the right role.
The AI-specific failure modes
| Risk | What it looks like | Control |
|---|---|---|
| Hallucinated schema | The model queries columns or tables that don't exist | Ground every prompt in live introspection; validate identifiers; self-correct on execution error |
| Data leakage | A query returns rows the asker isn't entitled to | Scope from the session into both prompt and query; let the BI engine's row-level security be the source of truth; mask denials so they don't reveal existence |
| Destructive or injected SQL | Generated or coerced query mutates data | Allow only read statements; reject anything else; parameterise values; never let user text become an instruction |
| Runaway cost / latency | Every question hits the model and scans large data | The cascade — exact/template/cube/semantic — before the model; cap result sizes; statement timeouts; log token/latency/cost per call |
| Confidently wrong answers | A valid-looking query that misreads intent | Result-shape and intent checks; prefer human-validated examples; surface the query for inspection; capture feedback |
| Stale flywheel | A bad answer gets memorised and reused | Require validation before an entry is weighted highly; let humans flag/curate; isolate embeddings by model version |
Trust, scope, and visibility
The user's access scope is read from the verified session and woven into both the prompt (so generated queries are pre-filtered) and the executed query (so they are enforced regardless of what the model produced). Defense-in-depth: even a perfectly malicious query cannot widen the caller's reach.
- Classify data by sensitivity tier and gate access by role; on denial, mask the response so it doesn't even confirm that matching data exists.
- Keep embeddings isolated by the model that produced them, so swapping embedding models doesn't silently contaminate similarity results.
- Instrument everything: log which tier resolved each question, token usage, latency and an estimated cost per model call — this is how you prove the cascade is earning its keep and catch drift early.
- Run the model behind the provider-agnostic gateway from Blueprint 03 so you can choose hosted, local or sovereign models per deployment and A/B compare them on cost and quality.
When to reach for it — and when not to
Use it when
- Non-technical users need answers from data and a BI team is the bottleneck.
- Question volume is high and repetitive enough that caching and templates pay off.
- Data is sensitive and per-user scoping is mandatory.
- You can ground answers in a real schema or document corpus and define what "correct" means.
Avoid / defer when
- A handful of fixed dashboards would serve users better than open-ended querying.
- The data model is too volatile or undocumented to ground prompts reliably.
- Answers feed automated decisions with no human in the loop and no tolerance for error.
- You can't yet enforce per-user data permissions — fix that before exposing NL query.
Want this pattern applied to your build?
These are the structures we reach for. Tell us the system you're trying to build.