Episode six of our “Crossing the AI Chasm” focuses on driving success through metrics and objective analysis. As a recap, in episodes one through five of the series we've discussed the importance of a methodical and diligent approach to rolling out GenAI, and for that matter, any new technology. There is no need to be "slow and tedious", but rather companies must be "thoughtful" and "prudent" on how they implement new technologies; and metrics are the voice of truth that is needed to ensure the plan is either progressing as planned, or modified in time so that it does.
Once you've launched GenAI, and it is up and running in your organization, the real work begins. Continuous monitoring of GenAI's performance is essential to understand how well it's integrating with your business processes, and if in fact you are realizing the return you expected from your investment.
Below are five recommendations on how to use metrics and data to gauge the success of your AI deployment AND make decisions accordingly. Let's check them out!
Incorporate / Map GenAI into your KPIs and OKR
Avoid developing KPIs and OKRs that are all about GenAI. If AI has been well integrated to your business it should be "imperceptible" to the operators and users. It should just work. Similarly, the corporate KPIs and OKRs should be just that - the metrics used to measure the company's success; and not the metrics of GenAI utilization by itself.
Look at the KPIs that you have already established, and link the roll-out of your GenAI applications to those. With a baseline determined, and holding other variable constant, the changes observed should be attributed, wholly or in part, to your GenAI applications.
Measure relentlessly
Nearly every application in the marketplace collects data broadly and automatically. As you set your KPIs, ensure that you will not create a burden on your users and operators by asking them to collect data. That said, if you identify an element/factor of your KPIs as a "manual" input, make sure that you give them the tool(s) to collect the data manually, and preferably - easily.
That said make sure that you collect your data set relentlessly, accurately, and consistently. The data will hold the truth and not yield clear and reliable answers unless you take all the measurements, at the same time, under the same conditions, etc.
As a shameless plug for the users - do it right, and collect the data as a byproduct of the work itself, not as new and additional work or the team. If you are asking people to go to a Google file to post an entry, or asking them to send an email when something has to be documented, you have other issues to address.
Measure your corporate adoption
Another key element in collecting the data is making sure that the sample group is as broad as possible, and that is driven by your team's adoption. As you measure and collect data, look at factors like:
Workload vs. on-time work completion
Given constant work, are we getting more or less work done on-time?
Has the workload capacity increased?
Variation in hourly employee and W2 employees hours worked.
Don't sleep on collecting hourly work data from your W2s. They are can represent the majority of the workforce benefitting from GenAI tools (Sales, Marketing, Finance, etc.)
User feedback - Customer and internal users
Is our quality of work better or worse? If adding AI to the production line is making the complaints soar, then there is an element of "Qualitative" analysis that need be done; but it starts with collecting the number of complaints or praises received and that can be linked to AI implementation.
Percent work-product done by AI vs. "Human Only"
In all cases, establishing a baseline is critical. As well, and as possible, it is imperative to control other variables so that the change can be attributed as closely as possible to the AI feature or software that was implemented.
Analyze Feedback and Performance Data
With feedback and performance data in hand, the next step is to analyze this information to identify trends, patterns, and anomalies.
This analysis will help you understand the strengths and weaknesses of your GenAI solutions. Are there certain tasks that GenAI excels at, while others need improvement? Are customers reporting a better experience, or are there new challenges that have arisen since GenAI's implementation?
Make Iterative Improvements
Armed with insights from your analysis, you can begin making iterative improvements to your GenAI systems. This might involve tweaking algorithms, expanding datasets, or even retraining the AI with new information.
The goal is to address any shortcomings and enhance the areas where GenAI is already performing well. Remember, the field of AI is constantly evolving, and your GenAI solutions should evolve with it.
PS - It's new technology - so make sure you use an integration approach that allows you to change AI components with ease!! When the data tells you to, make the changes accordingly!
Crossing the GenAI chasm is a "doable do" if we go through the blocking and tackling; and we monitor for progress as expected. Launching GenAI in your business is a long term process. Plan to measure its progress so that you can adjust and steer the plan towards success.