The Dashboard’s New Job Is To Keep AI Honest
Stu Sjouwerman is co-founder and CEO of ReadingMinds, a pioneering AI-moderated interview platform for conducting sentiment analysis.
Dashboards have been a trusted business management tool for decades. Their use was shaped, in part, by the measurement framework that Robert Kaplan and David Norton introduced in their book, The Balanced Scorecard. The framework offered a structured way for companies to identify and track the most important metrics in running a business.
That use is being challenged and complicated these days, though, as AI agents have emerged in a big way. Instead of executives being the sole consumers of dashboards and reports, AI agents are increasingly being introduced to support or automate portions of the decision-making process. That could potentially spell an obvious boon to productivity and agility, but it can also prove a risky proposition. After all, many organizations are still at the early stages of AI adoption, not scrutinizing the quality of AI outputs.
Dashboards are not dead, and they aren't going away (at least not any time soon). Rather, they have an important role to play in keeping a “human in the loop” as AI is deployed in new and autonomous ways.
Business intelligence software has long been based on the same replicable process: gather data, create a report, review the report, make decisions, rinse and repeat.
That process was designed for humans, though, not AI agents. Adding AI agents into the mix changes things. AI agents typically don’t interact with dashboards the way humans do; they rely on direct access to underlying data, events and system outputs to assess a situation, update a record or route customer interactions: processes that can transpire in seconds.
It’s the difference between the traditional process of informing humans and the rapidly emerging process of informing AI agents. Most organizations aren’t quite there yet. Getting there requires organizations to rethink not just how work is executed, but how AI performance itself is evaluated.
The focus of most executive attention when it comes to AI investments is on productivity and efficiency. Are these tools saving time? Are they streamlining operations? Are they seeing a positive impact on labor? The Wall Street Journal reported in late 2025 that 47% of marketing executives expected to be eliminating staff; some were already doing so.
Certainly, these measures matter to businesses, but they’re arguably not the most important. As Gartner noted, while 65% of chief marketing officers (CMOs) think advances in AI will have a significant impact on their jobs, only 5% of those using GenAI can point to significant business gains. What this reveals is a tendency to measure what’s easiest to quantify rather than what actually drives business outcomes.
In an agentic environment, that means focusing less on activity and efficiency and more on decision quality. Are AI agents consistently making the right decisions? The answer, unfortunately, is “not always,” and that can be a problem.
There are some well-documented examples of how even big brands have fallen prey to faulty AI agent decision-making. In 2024, Air Canada's AI agent gave a passenger incorrect information about bereavement fares. New York City’s AI chatbot MyCity reportedly advised employers that they could pocket workers’ tips and told landlords they are not required to accept Section 8 housing vouchers. Zillow’s Offers program relied on pricing models that contributed to the overvaluation of homes, resulting in a loss of $881 million to the company.
The problem with AI agents making mistakes (versus people) is that AI agents make those mistakes at scale across potentially thousands of interactions, impacting connected systems within the enterprise.
It’s a real and growing problem. According to Sinch’s AI Production Paradox report, 74% of enterprises have already rolled back their use of AI agents due to governance failures. Dashboards aren’t the right tool for detecting or monitoring these types of missteps.
In agentic environments, there are three critical keys to ensuring reliable oversight: provenance, decision quality metrics and structured inputs.
2. Decision quality metrics, as we’ve alluded to, are metrics that aren’t focused on outputs, but on the value of the decisions being made. Did the agent’s decision lead to the correct output?
3. Structured inputs should be machine-readable data that goes beyond narrative reports or summaries.
The bottom line and an important question to ask is, “Can your agents’ actions be audited?” If not, you’re not applying the requisite oversight to ensure that decisions don’t go wrong.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?