Dashboards are not failing. They are unreachable.
BARC's BI & Analytics Survey 26, published in November 2025 from more than a thousand professionals worldwide, reports that on average 25% of employees use BI software in their daily work. In large enterprises that drops to 16%. In smaller companies it reaches 44%.
The same survey finds that the people who do use it are getting value: 97% report faster reporting or planning, 96% better data quality, 94% better decisions.
Put those two findings next to each other and the problem changes shape. The tools work. They reach one person in six. Another dashboard, a cleaner layout or a better chart library does nothing about the other five, because those five were never going to open a dashboard. They have a question, occasionally, and the cost of getting an answer — find the right report, work out which filter applies, ask an analyst, wait two days — exceeds the value of knowing.
Why "just ask your data" is the right idea
The pitch is genuinely good, and it addresses the reach problem directly rather than decorating around it. A regional manager types "why were returns up in the north last month" and gets an answer, a number, and the query that produced it. No report to find, no analyst queue, no dashboard to learn.
It also removes the thing that actually kills dashboard adoption, which is not complexity but pre-definition. A dashboard can only answer questions someone anticipated when they built it. Every real question is slightly off the grid, and the moment it is, the person is back to asking a human.
So the ambition is sound. The failure is in the middle, and it is not where the demos suggest.
What happens when the schema is real
Spider 2.0 is a benchmark built to test language models on actual enterprise text-to-SQL workflows — 632 real problems spanning BigQuery, Snowflake, PostgreSQL, DuckDB, ClickHouse and SQLite, some requiring project-level DBT work. The best system evaluated, o1-preview running in an agent framework, completed 21.36% of tasks.
For comparison, the same class of approach scores 91.2% on the original Spider benchmark and 73.0% on BIRD. The chart at the top of this article is those three numbers.
The paper is explicit about what changed, and none of it is about model intelligence:
- Databases average 743.5 columns, against 54.2 for BIRD. Finding the right column is most of the task.
- Correct answers average 148.3 tokens of SQL against 30.9, and frequently run past a hundred lines.
- 85.98% of problems need dialect-specific functions — Snowflake is not BigQuery is not ClickHouse.
- Solving them requires reading codebases and documentation and running several exploratory queries, not emitting one statement.
Spider 2.0 was published in March 2025 and frontier models have improved considerably since. Quoting 21.3% as a current capability would be dishonest, and any vendor showing you a 2025 benchmark as though it describes 2026 systems is arguing rather than measuring. What has not changed is the cause: your warehouse still has hundreds of columns per table, your dialect still has its quirks, and the gap between a teaching schema and a production one is structural. Absolute scores move. The reason for the cliff does not.
The problem the benchmark cannot measure
Here is the part that should worry you more than the accuracy figure, because no benchmark captures it.
Spider 2.0 has a correct answer for every question. Your business does not. Ask four people what "revenue" means and you will get recognised revenue, booked revenue, revenue net of refunds, and revenue excluding intercompany. Ask what an "active customer" is and you will get thirty days, ninety days, any login, any transaction, and "not churned". These are not edge cases. They are the normal state of an organisation that has grown.
An agent pointed at raw tables will resolve that ambiguity by guessing — competently, invisibly, and differently each time. It will write syntactically perfect SQL, return a confident number, and that number will disagree with the one in the board pack. Somebody will act on it.
This is strictly worse than no answer. A dashboard that does not exist sends you to an analyst. An agent that answers wrongly sends you to a decision.
The semantic layer is the actual product
Which is why the engineering work in agentic analytics is almost entirely upstream of the agent. The deliverable is a governed set of definitions — metrics, dimensions, entities, joins, filters — that the agent queries through rather than around.
- → Picks join paths by guessing, and a wrong join inflates numbers silently
- → Invents a definition of the metric on every run, so the same question gives different answers
- → Has no way to know a column is deprecated, test data, or soft-deleted rows
- → Cannot be audited, because the reasoning is regenerated each time
- → Joins are pre-declared, so the agent selects metrics rather than inventing SQL
- → One definition of revenue, owned by someone, versioned, and the same on every run
- → Row-level permissions are enforced below the agent, not prompted into it
- → Every answer traces to a named metric definition a human can inspect
This reframes the project. "Add an AI layer to our BI" is mostly a data modelling and governance programme with an interface on the end. That is less exciting on a slide and it is where the work genuinely is — and if your warehouse is not in a fit state for it, that question comes before this one. We covered the architecture choices in data lake vs warehouse vs lakehouse.
An analytics agent has to be able to say no
The single most underbuilt feature in this category is refusal. A system that always produces a number is a system that produces wrong numbers, because some questions cannot be answered from the data available and the honest response is to say so.
How to know whether it works
The same discipline that governs any production AI system applies here, and this category tends to skip it because a plausible-looking number feels like success in a way that a plausible-looking paragraph does not.
Build a question set before building the agent. A hundred or so real questions — from the analyst queue, from Slack, from the people who currently ask a human — each with a verified correct answer. Then measure three things separately, because they fail for different reasons:
- Did it answer correctly? Against the known value, not against whether the SQL ran.
- Did it refuse when it should have? Include unanswerable and ambiguous questions in the set on purpose. A system that scores well on answerable questions and never refuses is not a good system.
- Did it answer when it should have? Over-refusal makes it useless and is the failure mode of a system tuned only on the first two.
Track these by question type rather than as one headline number. An aggregate hides the category where it is failing, and that category is usually the one that matters most.
What to build, in order
Collect the real questions
Two weeks of what people actually ask analysts, in their own words. This is your requirements document, your evaluation set and your scope boundary at once, and it will not resemble what the dashboards currently show.
Define the twenty metrics that matter
Not every metric. The twenty that the questions keep touching. Each one gets a written definition, a named owner, and agreement from the people who will dispute it later. Expect this step to surface disagreements that predate the project.
Build the semantic layer for those twenty
Declared joins, enforced permissions, deprecated columns excluded. Narrow and correct beats broad and approximate — the first wrong answer in front of an executive costs more trust than six months of coverage buys.
Add the agent, with refusal and citations
Only now. Every answer shows the metric definition and the generated query. Measure against the question set, including the questions it should decline, and expand coverage on evidence rather than on enthusiasm.
People who have never opened the BI tool are asking questions and getting answers they trust enough to act on. Every answer is traceable to a definition someone owns. The system declines a meaningful share of questions and nobody minds, because the ones it answers are right. And the semantic layer you built is now powering the dashboards too — which is the part that outlasts whichever agent framework you chose this year.
Frequently asked questions
Thinking About Letting People Ask Your Data Questions?
We start with the questions people already ask and the definitions nobody has agreed, because that is what decides whether the rest works. Book a free 30-minute call with our data engineering team. NDA first, no obligation.