The importance of tracking impact
Nearly any chief digital /data officer will tell you that measuring the business impact of AI/genAI use case is critical – for securing funds, assessing the effectiveness, and making smart strategic decisions about future technology investments. But a McKinsey survey found that only 10% of organizations said they implement regular and systematic tracking of the value of their AI/genAI initiatives.
“Despite recent excitement and investment in AI/genAI, most organizations struggle to measure their impact, due to a lack of performance management infrastructure,” said Matt Fitzpatrick, a senior partner at QuantumBlack, AI by McKinsey. “They have new data warehouses, models, and chatbots, and they have faith that their efforts are working. But they lack a clear methodology to quantify and monitor their effectiveness. Organizations must develop this capability.”
There are a number of challenges that make it difficult for companies to do so. They include:
- Stakeholder alignment and collaboration
- Defining meaningful metrics
- Isolating monetary impact is hard and complex
- Data collection and integration
- Data accuracy and reliability
And yet some organizations are beginning to figure it out. What do they do differently and what can the rest learn from them?
First, they differentiate between model performance and business performance. Both are important but inform different needs of the overall business. The table below shows how:
|
Category |
Model performance |
Business Impact |
|
Purpose |
Guide model tuning direction and ensure the model has sufficient predicting or classification power. |
Checks if AI/gen AI project is delivering impact (eg, sales uplift) to business. |
|
Metrics |
Statistical metrics such as MSE (mean square error) for regression models or roc-auc for classification models. In most cases, the metrics will be linked to the relevant business metric. |
Often uses business metric – such as sales or customer growth – as a tracking metric. Impact metrics tend to be more important than model metrics. |
|
Methodology |
Standard methodology exists. Cross validation works at most times. |
Highly dependent on business use case. Typical methods include tracking of leading indicators and A/B testing. |
|
Baseline |
Often uses less complex model or un-tuned model as a baseline. |
Depending on methodology, could be sales before adopting analytics or comparing adherence group to non-adherence group. |
|
Timeline |
Need to revisit model performance regularly (eg, monthly) to ensure model parameters and input is not outdated. |
Depends on business case. But only can start after AI/gen AI project launch. |
Model performance tracking is a staple of the ML lifecycle, and most organisations get it right. However top performers know how to focus on business impact tracking, as well. The most effective organizations take a four-step approach:
-
Define and align on leading and lagging KPIs
The first step in measuring AI/genAI impact is to clearly define the key performance indicators (KPIs) that will determine the use-case’s success. These KPIs should be established early in the project to set clear expectations and ensure that all stakeholders are on the same page.
KPIs can be categorized into two types: leading and lagging.
- Leading KPIs are forward-looking metrics that predict future events or trends. They serve as early indicators, helping leaders anticipate changes and gauge the potential success of the AI initiative.
- Lagging KPIs reflect past events or outcomes. They provide feedback on what has already happened, confirming trends and the overall effectiveness of the AI application.
Consider, for instance, a hyper-personalization project aimed at enhancing marketing campaigns. In this use case, a leading indicator might be the open and click-through rates of SMS or email communications. These metrics can be tracked in real-time, providing immediate insights into the campaign’s performance.
A lagging indicator might be the subsequent uplift in revenue or the increase in customer lifetime value, which are typically assessed after the campaign has been fully executed. Defining and measuring both types of KPIs is essential to successfully measuring use-case impact.
-
Ensure the feasibility of chosen indicators
Once the KPIs are defined, it is crucial to assess how feasible it is to measure these indicators effectively. This step involves ensuring that the necessary data can be collected, and that the metrics can be accurately tracked without significant obstacles.
Returning to the previous use-case example, the organization would need to confirm that open and click-through rates can be extracted from relevant systems. This might require collaboration with the marketing team to implement appropriate tracking methods, such as using tracking pixels in emails.
Several factors need to be considered during this phase:
- Frequency of data collection: How often will these metrics be collected and analyzed?
- Data processing: How will the data need to be processed to ensure its accuracy and relevance?
- Data quality: Are there any data-quality issues that need to be addressed?
It’s also a good idea to conduct tests to identify any potential gaps in the data-collection process and to rectify these issues before the project progresses too far.
-
Automate and integrate into a portfolio to track value rigorously
After ensuring the feasibility of the chosen indicators, the next step is to automate the data-collection process. This is crucial because manual data collection can be time-consuming and prone to errors, detracting from the efficiency and scalability of the AI/genAI project.
Key considerations for this step include:
- Deployment: Where will the metric-collection solution be deployed?
- Data security: How will data security be ensured throughout the process?
- Maintenance: Who will be responsible for maintaining the solution?
- Results presentation: How and where will the results be displayed? (A business intelligence tool might be an appropriate choice for visualizing and sharing the data.)
Automating these processes allows organizations to rigorously track the value generated by their AI/genAI initiatives, ensuring that the benefits are consistently measured and reported.
-
Communicate the impact
Finally, the impact of the AI/genAI project needs to be communicated effectively to stakeholders and the broader organization. Without proper communication, the value of the project may go unnoticed, undermining its success.
Throughout this process, it’s important to involve departments other than IT; finance executives, for example, should be brought in to help quantify the impact in monetary terms where possible. Additionally, providing stakeholders with access to a live dashboard can be highly beneficial. This enables real-time tracking of the AI/genAI project’s impact, which can be used to inform regular status meetings and strategic discussions.
The bottom line: measuring the impact of AI/genAI initiatives requires a thoughtful, structured approach. By defining and aligning on KPIs, ensuring their feasibility, automating data collection, and effectively communicating results, organizations can maximize the value of their AI/genAI investments and drive meaningful business outcomes.
At QuantumBlack Labs, we specialize in helping organizations scale AI. Email Matt Fitzpatrick to start scaling, measuring, and optimising your AI/genAI investments
The authors would like to thank Mohammed ElNabawy and Rohit Rathod for their contributions to this article.
DataIQ members can gauge their AI capabilities through exclusive assessments.
