“‘Garbage in, garbage out’ isn’t the worst thing in the world, but putting trust in that garbage is,” warned Gartner vice-president analyst Carlie Idoine during the keynote at the Gartner Data & Analytics Summit in Sydney this month.

She was joined by fellow vice-president analyst Gareth Herschel, who highlighted a 2024 Gartner survey which found that data availability and quality were the top obstacles to implementing artificial intelligence (AI), adding that “if you cannot trust the data, you cannot trust the AI that uses it.”

While governance is key to trustworthy data, Herschel said it is not practical to achieve fully governed data before delivering AI-powered capabilities. The answer, he suggested, is to implement trust models that rate the trustworthiness of data based on its lineage and curation, which can significantly reduce the risk of people using incorrect data.

A key theme of the summit was the distinction between different forms of AI. Erick Brethenoux, chief of AI research and distinguished vice-president analyst, pointed out that generative AI and AI agents “have nothing to do with each other”.

AI agents, he said, have been around for at least 30 years and used for tasks like predictive machinery maintenance, whereas agentic AI is primarily a marketing term. Vendors tend to conflate the two concepts because of the significant revenue potential in generative AI, but Brethenoux said “it’s important to name things the right way.”

“AI agents can use models, like large language models [LLMs], or not,” he said, but combining the two can yield interesting results. However, because generative AI is non-deterministic, meaning the same prompt can yield different responses, it is impossible to rely on traditional testing. Instead, organisations need to place guardrails around the model and run simulations to ensure it behaves as intended.

One of the main advantages of agents is that they only consume resources when active, making them a faster and more cost-effective way of implementing multi-step processes in parallel. For example, if a loan application involves multiple checks, each can be allocated to a separate agent. The process can then be terminated as soon as any one check fails, rather than waiting for all of them to complete sequentially.

On AI working in the background without necessarily having a human to make the final decision, Brethenoux said where reaction time is important or delaying a decision increases risk, it might be appropriate to allow a system to respond automatically. This also applies where risks are low – for example, an agent could automatically make travel arrangements based on a user’s past preferences. In other situations, it is better to have a human check a proposed action before it is implemented.

“Autonomy is one of the most sticky problems we have with software agents,” Brethenoux observed, noting that while people are comfortable receiving advice from software, they are still adjusting to autonomous action.

Return on investment The economics of AI, however, are not as simple as they might seem, according to Gartner vice-president analyst Luke Ellery. Citing Microsoft’s figures, he noted that Copilot saves the average employee only 14 minutes a day. Once productivity leakage is factored in, this is worth around $800 a year, while the all-up cost of Copilot is around $1,150. The benefit, Ellery explained, lies elsewhere. People using Copilot daily perceive themselves as 2.5 times more productive, and their employee net promoter score (NPS) is 59, compared with an average of 21. “NPS is a hard value, but it is not a financial value,” he noted. Ellery said that while returns on investment is appropriate for evaluating generative AI use cases that extend current capabilities, it is not the whole story. Using generative AI to fundamentally change a business model is a more complex situation that often involves several simultaneous, multi-million-dollar investments over a long-term horizon, which should be viewed as a bet on the future. Consequently, he recommended that use cases should be categorised by the type of value they create, expectations should be set carefully with stakeholders, and organisations should build a portfolio of projects that collectively match their desired outcomes.