The Traditional ROI Measure is not for AI
Dr. Caroline Chibelushi, KTM Artificial Intelligence, shares her thoughts on the challenges of measuring the return on investment for AI.
Dr. Caroline Chibelushi, KTM Artificial Intelligence, shares her thoughts on the challenges of measuring the return on investment for AI.
Introduction
The adoption of Artificial Intelligence in business continues at an exponential rate, with major research organisations predicting that $77 billion will be spent globally on AI tools for automation, cognition and virtual agent solutions by 2022 (IDC). 60% of occupations will be automated, implying substantial workplace transformations and changes for all workers by 2030 (McKinsey), and 25% of all digital workers will be using some form of conversational virtual assistant by 2022 (Gartner), with financial and insurance sectors leading the way (IPSoft).
However, many companies still wonder where exactly AI could create value or how to get started. Many questions remain, even among advanced adopters of AI technology. How can companies apply AI to empower employees, engage with customers or transform their business? Where do the benefits lie, and what are their blockers? (AI in Europe 1, AI in Europe 2).
All businesses that are considering adopting AI need to make a solid business case for AI investment. AI projects are exploratory in nature and not similar to IT or software solutions. Fagella mentions that “in reality, identifying a metric to reliably measure the impact AI is having at a business is very hard”, and this is exacerbated by the fact that it is also very complex to estimate the financial investment or costs associated with the implementation of AI (covering not only the one-time direct costs, but also the ongoing operational, and downstream costs).
Measuring the return on investment for AI may differ between organisations and it is heavily dependent on their hurdle rates for their investment, data and circumstances. That is, each organisation’s AI project will often have its own context and data – this data will be used to train, test and refine the AI model. In most cases, it will also define the ROI measure for the AI project, so this brings us to the question: how can any organisation be able to quantify the impact of AI without exploring their data, running a pilot study and understanding the details of the project’s long-term investment? To my understanding, this is impossible, and any estimation done without investing in data exploration and a pilot study is deemed to fail.
The Return on Investment
The return on investment (ROI) is a tool traditionally used in the private sector to evaluate and compare projects and investments. Simply put, it is the net earnings from a project divided by the project costs.
ROI returns are perceived as the monetised benefits of a project. It can be related to cost-benefit analysis and cost-effectiveness analysis of a project; however, the reporting of the relationship between benefits and costs are often customised for each organisation. ROI measures help make decisions about where to invest and identifying the project with the greatest returns as a percentage of costs. It can be used to evaluate projects according to a profitability metric or to improve them to generate higher net returns. While this is the case, its simplicity can prevent the consideration of other important features of projects that are not included in the ROI calculation, and should thus be only one of the evaluation determinants.
The Return on Investment of AI
Preparing business cases that allow a smooth adoption of AI within an organisation is, without a doubt, a challenge. One of the reasons for this includes the complexity and lack of capabilities to define the ROI of AI measure for their project. As a result, many businesses delay the investment and the process of adopting AI and fall in a risk of being less productive or losing their competitive advantage.
Since predicting how artificial intelligence is going to produce a return on investment is such a complex matter, the question to be asked is what kind of ROI considerations can be made?
What can decision makers do when it comes to thinking about AI and ROI?
1. Have a clear definition of the problem and investment resources
Symrise, a global fragrance company in Germany, used AI to produce new perfumes for Brazil’s millennial market, boosting their revenues and their global reach; however, it took the company two years to get to that point, with most of the time spent in understanding the details of the problem to be solved, intensive training of the AI system by their perfumers, as well as integrating and upgrading the existing IT system to allow companies disparate data to be linked to the AI system (Shacklett, 2019).
Symrise took the approach of thinking less in terms of the technology, and more about the impact the company wanted AI to have on the people connected to their business – both customers and employees. Decision makers need to consider a human-centered AI – an approach that focuses on augmenting the workforce to improve customer and employee experiences (Reich, 2018). This requires spending more time to continually recalibrate and train the AI tool because the human mind (and emulating it) can be unpredictable.
2. Have a clear understanding of the data, returns and hurdle rates for the investment
I will be using an example of a Lung Nodule Flagger (LNF), an AI tool that analyses the radiological images and identifies suspected lung nodules, notifying the radiologist directly. LNF has been trained on a dataset of prior images which human radiologists characterised in order to allow LNF to identify lung nodules similarly or more accurately than a human radiologist. The amount of disability-adjusted life years (DALYs) lost by the population for which the lung nodule flagger was used can be compared to the number of DALYs lost by the population for which it was not used. Through the use of the LNF, there is a reduction in DALYs lost – in this case, one could argue there is an economic value rather than a direct return on investment.
Additionally, LNF can change the rate at which lab tests and biopsies are conducted, and the rate at which the patients receive these tests. This means that it will also change the rate of conducting surgeries, chemotherapies and other procedures to patients.
“….AI enables patients to be more likely to receive diagnoses before they progress to be symptomatic, there may be a reduction in the utilisation of downstream services” (Powell, 2020).
These are benefits which cannot be measured through a traditional ROI procedure. While the impact of AI is quantifiable in this example, the estimates of its financial ROI are not portable in this context (Powell, 2020). Hence it is very important for decision makers to understand their data, and the outcomes which could come out of training AI with it.
To sum up, AI can be used to reduce costs, improve customer experience, and increase the productivity of a specific team within the business. The first step to understanding ROI in the context of AI could be to associate the returns with any type of positive business outcomes, not only those related to financial gains, like the indirect social and economic returns, and but also, those associated with enhancing a teams’ AI-related skillset.
3. Take a staged approach
Some AI applications link neatly to projected returns, making ROI calculations straightforward. For example, an energy producer could directly tie its investment in an AI-powered predictive maintenance tool to increases in equipment uptime, or reductions in maintenance costs. Other applications are more complex and unpredictable, making it challenging to use typical ROI approaches. To what extent, for instance, could reductions in crime be tied to AI projects?
Decision makers need to identify a pilot project with measurable metrics before AI is applied.
Alternatively, developing an AI solution for an existing business problem may be easier in terms of measuring success, rather than developing a completely new AI use-case with no precedent. Also, they need to identify means to run a pilot test and understand how the chosen pilot project helps the business gain knowledge of working with data and AI capabilities and its long-term benefits.
For example, Avanade has been working with an insurance company that began by automating rote, manual processes to prioritise claims and assign them to workflows. The company took the next step by adding cognitive services that could read all customer query tickets and understand the customers’ intent. The queries were then categorised and assigned to the right workflow. By using AI to answer more standard questions, the company liberated people to answer more complex questions, resulting in up to 60 % gains in efficiency and resolution time.
This example shows there is no need to take big steps to see positive results from AI implementations, but there is a need to understand its long-term benefits. A manager should find the right process prior to starting the project, then evolve their approach to deliver business results (Reich, 2018).
Conclusions
On the topic of measuring the return on investment of AI and its complexity, Roe (2018) says: “Cognitive AI systems are designed to magnify human talent, providing actionable information faster, reducing risk and identifying opportunities. In our view, the real potential of AI is a symbiotic relationship with people, almost like an assistant that enables humans to apply their attention, experience, and passions to solving problems that truly matter.”
As a result, these measurements are very much dependent on where the company is in its journey, its experience with implementing AI technologies, and on what kind of business value a company wants to achieve (is the company trying to enhance efficiency, reduce costs, reduce waste, ensure workers safety, improve the customer or employee experience?).
Internal stakeholders need to embrace the fact that AI projects are experimental and require reasonable investment on both time and financial resources. Instead of asking the question “what is the ROI of AI?”, they need to explore their data, understand the problem and run AI pilot projects with clear testing procedures and the means to measure the outcomes. Dedicating enough time and budget to conduct as many experiments as possible is also to be taken into consideration.
My advice is that business leaders should not explore the impact AI will have on their business without taking a few initial steps in understanding and experimenting with their data, as it contains knowledge which may be able to clarify questions related to AI’s ROI (e.g. understanding the clear problem that requires an AI solution, or exploring the appropriate tests to be applied on a project). In this context, missing data may negatively impact the successful adoption of AI, and could make a company less willing to invest in this technology in the future – when that investment stops, companies also lose the competitive advantage that AI is bringing to their business.
Artificial intelligence is a tool just like a laptop or a smart phone, that aims to assist humans in efficiently solving problems. Similar to AI, the benefits of a laptop cannot be measured through a traditional ROI procedure; however, it is easy to feel the negative impact on one’s productivity when a laptop is not used for daily business activities.
