Home ›Insights ›Data & Analytics
Data & AnalyticsAI EngineeringNovember 21, 202613 min read

Agentic Analytics: The Dashboard Was Never the Bottleneck

In large enterprises, 16% of employees use BI software in their daily work — while the people who do use it report that it helps. That is not a dashboard quality problem, and a better dashboard will not fix it. Letting people ask questions in their own words is the right idea. What stops it working in production is not the model: it is that nobody wrote down what "revenue" means.

VE
Vikgol Engineering Team
Data & AI Engineering · Vikgol
Share

Dashboards are not failing. They are unreachable.

BARC's BI & Analytics Survey 26, published in November 2025 from more than a thousand professionals worldwide, reports that on average 25% of employees use BI software in their daily work. In large enterprises that drops to 16%. In smaller companies it reaches 44%.

The same survey finds that the people who do use it are getting value: 97% report faster reporting or planning, 96% better data quality, 94% better decisions.

Put those two findings next to each other and the problem changes shape. The tools work. They reach one person in six. Another dashboard, a cleaner layout or a better chart library does nothing about the other five, because those five were never going to open a dashboard. They have a question, occasionally, and the cost of getting an answer — find the right report, work out which filter applies, ask an analyst, wait two days — exceeds the value of knowing.

16%
Of employees in large enterprises use BI software daily
BARC BI & Analytics Survey 26, Nov 2025
94%
Of those who do use it report better decisions
BARC, same survey
21.3%
Best system's accuracy on real enterprise text-to-SQL tasks
Spider 2.0, arXiv:2411.07763

Why "just ask your data" is the right idea

The pitch is genuinely good, and it addresses the reach problem directly rather than decorating around it. A regional manager types "why were returns up in the north last month" and gets an answer, a number, and the query that produced it. No report to find, no analyst queue, no dashboard to learn.

It also removes the thing that actually kills dashboard adoption, which is not complexity but pre-definition. A dashboard can only answer questions someone anticipated when they built it. Every real question is slightly off the grid, and the moment it is, the person is back to asking a human.

So the ambition is sound. The failure is in the middle, and it is not where the demos suggest.

What happens when the schema is real

Spider 2.0 is a benchmark built to test language models on actual enterprise text-to-SQL workflows — 632 real problems spanning BigQuery, Snowflake, PostgreSQL, DuckDB, ClickHouse and SQLite, some requiring project-level DBT work. The best system evaluated, o1-preview running in an agent framework, completed 21.36% of tasks.

For comparison, the same class of approach scores 91.2% on the original Spider benchmark and 73.0% on BIRD. The chart at the top of this article is those three numbers.

The paper is explicit about what changed, and none of it is about model intelligence:

  • Databases average 743.5 columns, against 54.2 for BIRD. Finding the right column is most of the task.
  • Correct answers average 148.3 tokens of SQL against 30.9, and frequently run past a hundred lines.
  • 85.98% of problems need dialect-specific functions — Snowflake is not BigQuery is not ClickHouse.
  • Solving them requires reading codebases and documentation and running several exploratory queries, not emitting one statement.
Read that number with its date attached

Spider 2.0 was published in March 2025 and frontier models have improved considerably since. Quoting 21.3% as a current capability would be dishonest, and any vendor showing you a 2025 benchmark as though it describes 2026 systems is arguing rather than measuring. What has not changed is the cause: your warehouse still has hundreds of columns per table, your dialect still has its quirks, and the gap between a teaching schema and a production one is structural. Absolute scores move. The reason for the cliff does not.

The problem the benchmark cannot measure

Here is the part that should worry you more than the accuracy figure, because no benchmark captures it.

Spider 2.0 has a correct answer for every question. Your business does not. Ask four people what "revenue" means and you will get recognised revenue, booked revenue, revenue net of refunds, and revenue excluding intercompany. Ask what an "active customer" is and you will get thirty days, ninety days, any login, any transaction, and "not churned". These are not edge cases. They are the normal state of an organisation that has grown.

An agent pointed at raw tables will resolve that ambiguity by guessing — competently, invisibly, and differently each time. It will write syntactically perfect SQL, return a confident number, and that number will disagree with the one in the board pack. Somebody will act on it.

This is strictly worse than no answer. A dashboard that does not exist sends you to an analyst. An agent that answers wrongly sends you to a decision.

The semantic layer is the actual product

Which is why the engineering work in agentic analytics is almost entirely upstream of the agent. The deliverable is a governed set of definitions — metrics, dimensions, entities, joins, filters — that the agent queries through rather than around.

Agent on raw tables
Fast to demo, unsafe to trust
  • → Picks join paths by guessing, and a wrong join inflates numbers silently
  • → Invents a definition of the metric on every run, so the same question gives different answers
  • → Has no way to know a column is deprecated, test data, or soft-deleted rows
  • → Cannot be audited, because the reasoning is regenerated each time
Agent on a semantic layer
Slower to build, safe to use
  • → Joins are pre-declared, so the agent selects metrics rather than inventing SQL
  • → One definition of revenue, owned by someone, versioned, and the same on every run
  • → Row-level permissions are enforced below the agent, not prompted into it
  • → Every answer traces to a named metric definition a human can inspect

This reframes the project. "Add an AI layer to our BI" is mostly a data modelling and governance programme with an interface on the end. That is less exciting on a slide and it is where the work genuinely is — and if your warehouse is not in a fit state for it, that question comes before this one. We covered the architecture choices in data lake vs warehouse vs lakehouse.

An analytics agent has to be able to say no

The single most underbuilt feature in this category is refusal. A system that always produces a number is a system that produces wrong numbers, because some questions cannot be answered from the data available and the honest response is to say so.

01
The term is undefined
The question uses a metric that is not in the semantic layer. Correct behaviour is to say which term is unrecognised and offer the closest defined ones, not to approximate it.
02
The term is ambiguous
Three definitions of "active user" exist and the question does not pick one. Ask. A single clarifying question is cheaper than a wrong number that reaches a slide.
03
The data cannot support it
The question needs a dimension nobody captures, or a period before the data existed. Say that plainly, rather than returning a confidently incomplete answer.
04
The answer rests on too little
Four rows, or a segment below a disclosure threshold. Return the count alongside the figure, or withhold it. Small-sample conclusions are how analytics loses credibility.

How to know whether it works

The same discipline that governs any production AI system applies here, and this category tends to skip it because a plausible-looking number feels like success in a way that a plausible-looking paragraph does not.

Build a question set before building the agent. A hundred or so real questions — from the analyst queue, from Slack, from the people who currently ask a human — each with a verified correct answer. Then measure three things separately, because they fail for different reasons:

  • Did it answer correctly? Against the known value, not against whether the SQL ran.
  • Did it refuse when it should have? Include unanswerable and ambiguous questions in the set on purpose. A system that scores well on answerable questions and never refuses is not a good system.
  • Did it answer when it should have? Over-refusal makes it useless and is the failure mode of a system tuned only on the first two.

Track these by question type rather than as one headline number. An aggregate hides the category where it is failing, and that category is usually the one that matters most.

What to build, in order

1

Collect the real questions

Two weeks of what people actually ask analysts, in their own words. This is your requirements document, your evaluation set and your scope boundary at once, and it will not resemble what the dashboards currently show.

2

Define the twenty metrics that matter

Not every metric. The twenty that the questions keep touching. Each one gets a written definition, a named owner, and agreement from the people who will dispute it later. Expect this step to surface disagreements that predate the project.

3

Build the semantic layer for those twenty

Declared joins, enforced permissions, deprecated columns excluded. Narrow and correct beats broad and approximate — the first wrong answer in front of an executive costs more trust than six months of coverage buys.

4

Add the agent, with refusal and citations

Only now. Every answer shows the metric definition and the generated query. Measure against the question set, including the questions it should decline, and expand coverage on evidence rather than on enthusiasm.

What success looks like

People who have never opened the BI tool are asking questions and getting answers they trust enough to act on. Every answer is traceable to a definition someone owns. The system declines a meaningful share of questions and nobody minds, because the ones it answers are right. And the semantic layer you built is now powering the dashboards too — which is the part that outlasts whichever agent framework you chose this year.

Frequently asked questions

Does agentic analytics replace our BI tool?
No, and framing it that way usually kills the project. Dashboards remain the right interface for the monitoring case — the same figures, on a schedule, where regular users want to see them without typing anything. What agents address is the long tail of one-off questions from people who do not open the tool at all. In practice both end up reading from the same semantic layer, which is the real integration point and the reason this work pays off twice.
Can we skip the semantic layer and point an agent at the warehouse?
You can, and it will demo well. The failure is not syntax errors, which are visible, but silent disagreement — a confident number built on a guessed join or an invented definition of the metric, which nobody catches until it contradicts the board pack. Spider 2.0 suggests how hard even the mechanical part is on real schemas, at 21.3% for the best system on 632 enterprise tasks, and that benchmark has one correct answer per question, which your business does not.
How long does a semantic layer take to build?
For the twenty metrics that cover most questions, weeks rather than quarters — provided the definitions get agreed. That proviso is the whole timeline. The modelling work is routine; getting finance, sales and operations to accept one definition of revenue is the part that slips, and it slips for organisational reasons rather than technical ones. Start it in week one and treat it as a decision to be chaired, not a document to be written.
What accuracy should we require before rolling this out?
Set the bar against the current alternative rather than against a hundred per cent. The relevant comparison is the analyst who answers these questions today, including their turnaround time and their own error rate. What matters more than the headline figure is the shape of the errors: confident wrong answers are far more damaging than refusals, so weight your evaluation accordingly and measure refusal behaviour as a first-class metric rather than an afterthought.
How do we stop people seeing data they should not?
Enforce it below the agent, never inside the prompt. Row-level and column-level permissions belong in the semantic layer or the warehouse, applied to the query the agent generates, so that an instruction in the question cannot widen access. An agent told in its system prompt to respect permissions is a control that fails the first time someone phrases a question cleverly. The agent should be incapable of reading what the user cannot read.
Is this different from the AI features in our existing BI tool?
Often less different than the marketing implies, and that is not necessarily bad news. If your BI platform already has a governed semantic model and its natural-language feature queries through it, you may already hold most of what matters and the build is far smaller than you think. Check which one it does: a feature that generates SQL against raw tables carries the same risks as any other agent on raw tables, whoever ships it.

Thinking About Letting People Ask Your Data Questions?

We start with the questions people already ask and the definitions nobody has agreed, because that is what decides whether the rest works. Book a free 30-minute call with our data engineering team. NDA first, no obligation.

#AgenticAnalytics#SemanticLayer#TextToSQL#BusinessIntelligence#DataEngineering#AIAgents#DataGovernance#Vikgol
VE
Vikgol Engineering Team
Data & AI Engineering · Vikgol
The Vikgol engineering team has shipped 90+ AI, web and cloud projects for startups and enterprises across the US, UK, UAE and India. We are an OpenAI Select Partner. On analytics projects most of the effort goes into definitions and governance rather than models, because that is what decides whether anyone trusts the answer.
Available Now · 72-Hour POC

Ready to Ship Your AI Product?
Let’s Build It Together.

Senior engineers on demand. Working prototype in 72 hours. NDA before we discuss anything. 100% code ownership to you — no lock-in, ever.

70+
Senior engineers on staff
90+
Projects delivered globally
72h
Working POC guaranteed
5★
Client satisfaction rating