Why are we still ONLY measuring AI by how much time it saves and not by quality?

I’ve been thinking a lot recently about how we measure the success of AI implementation. So much of the conversation still centres around productivity.

• How many hours did we save?

• How much faster can someone complete a task?

• How many processes can we automate?

• How much more can one person produce?

• How much money does this save?

Those are important questions. But from what I’m seeing at New Icon and with our clients, they only capture part of the value.

I work closely with developers, designers and project teams, and I’ve noticed something interesting about how the benefits of AI show up differently across those disciplines.

For our developers, the productivity gains can be very tangible. AI can accelerate repetitive development tasks, support debugging, help explore approaches and allow developers to iterate more quickly. That increased speed also creates space for additional refinement. More time to question an implementation, test alternatives and improve the end result.

It’s that we can do better work.

AI gives us another layer of challenge:

• We can use it to interrogate a brief before it reaches the team.

• We can ask it to find the gaps in our thinking.

• We can explore an idea from the perspective of a customer, developer, stakeholder or someone completely unfamiliar.

• We can take a complicated problem and test different ways of structuring it.

• We can challenge assumptions before they become decisions.

• We can refine something repeatedly without the practical limitations that come with asking another person to review version 17 of the same idea.

The output isn’t necessarily more. Often, it’s simply more considered. Perhaps we’re missing a measurement. When organisations build the business case for AI, productivity is attractive because it’s easy to quantify.

If something used to take an hour and now takes 30 minutes, there’s a neat number you can put into a spreadsheet. Quality is much harder.

• How do you quantify a better decision?

• What’s the value of identifying an issue before development starts?

• How much is a clearer brief worth?

• What is the commercial impact of considering an alternative that otherwise wouldn’t have been explored?

• How do you measure the value of someone having the capacity to challenge their own thinking more thoroughly?

• Those benefits are less convenient to measure, but that doesn’t make them less valuable.

I think we need to look at AI implementation through at least three lenses:

• Efficiency | Can we achieve the same outcome with less time or effort?

• Quality | Can we achieve a better outcome with the same resources?

• Capability | Can we now do something that wasn’t realistically possible before?

The most interesting AI implementations, in my view, will increasingly touch all three. There’s also a risk in assuming that every hour AI saves should immediately become another hour of output. Sometimes the best use of that capacity is thinking. Give a developer more time to refine an implementation. Give a designer more opportunities to interrogate the user journey. Give a project manager more capacity to identify risks before they become problems. Give a leadership team the ability to explore five scenarios instead of one.

That’s still productivity, but the return isn’t necessarily volume. It’s quality. At New Icon , that’s increasingly how I’m thinking about AI adoption.

The question shouldn’t only be: “How much faster can AI make us?”

We should also be asking: “How much better can AI help us become?”

Because if we’re only measuring the hours AI gives back, I think we’re missing a significant part of its value.