The Last Mile: Is user adoption the Achilles' heel of AI?

Studies have shown that roughly only 20% of AI projects succeed and only 15% of analytics projects. But why?

May 03, 2023

Companies pump millions into AI projects and tools.

They oversee one crucial thing: the human factor. AI and analytics fail in user adoption. Salespeople barely glance at the created forecasts and dashboard. Marketers shrug off all the recommendations. Doctors hastily push away the AI diagnosis.

Orpheus was not just any ordinary musician, but one with a gift so divine that his playing could soothe even the fiercest of beasts and move the very rocks themselves.

His fame spread far and wide. He eventually fell in love with the beautiful Eurydice, whom he married in a joyous ceremony.

But their happiness was short-lived, for on the very day of their wedding, Eurydice was bitten by a venomous snake and died. Orpheus was heartbroken and inconsolable, and in his despair, he decided to venture into the underworld to retrieve his beloved wife.

Armed with only his lyre, Orpheus made his way through the dark and treacherous underworld, playing his music to ward off the beasts that threatened to devour him. He finally reached the throne of Hades, the ruler of the underworld, and begged for his wife's return.

Moved by Orpheus's music, Hades agreed to allow Eurydice to return to the world of the living, but on one condition - Orpheus was not to look back at her until they had both returned to the surface. Orpheus agreed and set off with his beloved wife.

But Hades played a devilish trick. On the last ascent out of hell, Hades blunted the sound of Eurydice’s footsteps. And when Orpheus finally reached the land of the living, he turned around to see if Eurydice was behind him.

In that moment, Eurydice was whisked back to the underworld, never to return to the world of the living again.

Like Orpheus, we stumble in the dark.

We engineers and data scientists love to tinker and fiddle with things until they're perfect. We'll spend hours collecting data, building pipelines, and constructing model architecture. And, we'll finally come up with a shiny new AI model that promises to revolutionize the world!

But here's the kicker, the real challenge comes when we try to get people to actually use it. That's the last mile. We've come so far, but we still need to step across the fold.

This is the Last Mile Problem of AI. It isn’t an issue of implementation. It is an issue of adoption.

The Traditional Way of Building AI

Imagine you are part of the data team of a major airline. The executives, driven by raging customers, were looking for ways to forecast delays and cancellations. So your team is tasked to develop a model for weather predictions.

You, eager to please, jump right into the project.

You gathered data from various sources, without really thinking about how they would use it. Data on temperature, humidity, wind speed, precipitation, and more, without any real sense of how these variables were related to flight operations.

Once you had gathered all this data, you started to set up a complex data infrastructure that aggregated and processed all of it. It took months of sweat to build a system that could handle massive amounts of data.

Finally, you were here. The fun part. You could start modeling. Your team jumped in, and spent countless hours tweaking the model, running it through test after test, until they were confident that it was accurate enough to be useful.

You packaged it all up in a nice API. Proud of your work, you delivered it and jumped on to the next project.

Months later, you went back to the API, only to discover: not a single call.

What went wrong?

Step-by-Step From the User to the AI

Imagine we're tasked with building the model for weather prediction again.

Let’s take a step back. We flip things around and consider our end result from the get-go.

Before we start the project, we should ask: What if we succeed? How would we use this model in the real world?

  1. Think about outputs, not outcomes. We are looking to enable better decisions, decrease costs, and increase revenue not have a high model accuracy.Driving questions: If I built this dashboard or model, what decision is taken with it, and what value is driven downstream?Go into the hypothetical extreme: If you would have perfect predictions, what decisions would and could you take? What value can you drive?Output: definition of the business value.How can we use weather prediction to enable better decisions, decrease costs, and increase revenue? We could forecast more timely whether flights have to be canceled or delayed. The airline is able to react faster and schedule a replacement flight avoiding the cost of cancellations (business value 1). It can also improve customer satisfaction by avoiding flight disruptions (business value 2).

  2. Once we've identified these outcomes, we can start thinking about how we're going to measure success.Driving questions: What are the quantitative and qualitative metrics we'll use to determine whether the model is driving the desired outcomes? How accurate do our predictions need to be to deliver the intended value?Output: metrics for the model quality, metrics for the delivered business value.We're aiming for nothing less than perfect predictions. To make it happen, we need to hit the sweet spot of accuracy, with at least 80% certainty, and timing. By timing, we mean that the predictions are delivered with enough time for airline operators to take action. We'll have to roll up our sleeves and interview our trusted teammates in operations to get a sense of how much time is needed in advance. Ultimately, we're after two types of measurable results: reduced cost and increased customer satisfaction. We'll measure cost reductions by tracking the refunds avoided thanks to our top-notch predictions. And when it comes to customer satisfaction, we'll use NPS, the gold standard metric for measuring customer loyalty and happiness.

  3. From there, we can start building an MVP. Deploy something asap and then look at how it drives the business value quantitatively and qualitatively.In the best case deploy something based on heuristics that codify rules. Otherwise, use simple techniques for prototyping. The accuracy gains are not worth the overhead and the loss in transparency. Focus on decision trees and regressions.Driving questions: Is the output actually used? Is data available to develop the model or prediction?Output: a simple model, model usage, business value captured.In our case, we experiment with select airports that frequently suffer from massive thunderstorms that can be forecasted with some accuracy. We’re providing the prediction to the operators for these airports, giving them a heads-up well in advance, so they can take proactive measures and avoid canceling flights. We track their refunds to see if the prediction actually reduces cost and capture customer voices to see if our measures improve customer satisfaction, making for a stress-free travel experience for everyone involved.

We have a simple model. We know whether the model is used or not. And we know whether the business value was captured. This gives us several lines of attack.

  • First, if our model isn't used, we need to dive deeper into the issue and find out why it doesn't fit into the operators' work. It's not their fault for not using it, it's our responsibility to build a tool that fits their needs. To solve this, we should work on building trust, integrating it more seamlessly, and positioning it better.

  • Second, if the business value wasn’t captured, think about whether a more accurate model would really have an impact. Most of the time this is not the case. Scrap the project and find something else.

  • Third, if the model is being used and we've captured business value, we've hit the Goldilocks zone. We can start thinking about spending manpower on developing a dashboard or model.

First Commandment of AI Projects

Remember: The technical implementation is secondary, the adoption is primary. If users don't adopt it, it's useless.

Sources

Reply

or to participate.