The Broken-Leg Problem: Why Users Hesitate to Trust AI Systems

Understanding the Psychological Barrier to Accepting Machine Learning Predictions

May 03, 2023

Do your users hesitate to use your AI? It is common for users to feel uneasy about relying on artificial intelligence. One reason for this is the broken-leg problem.

Let's say you have a model that can accurately predict whether a person will go to the movies in the next week. But what if that person just broke their leg and can't leave the house? The guy that knows that his buddy has broken his leg will always beat your model, no matter the accuracy.

This is an issue with statistical models: they can miss important contextual factors that affect the outcome.

The broken leg problem can be observed across industries and user types. Studies have shown it in manufacturing, healthcare, technology, for doctors, bluecoller workers, knowledge workers, and managers.

But users generalize this attribute on every AI as well. Its the user imagining an ommitted variable bias in their head.

Users often generalize this attribute to all AI. They imagine an omitted variable bias that could affect the accuracy of predictions. They believe that AI models cannot see their uniqueness, but are only making decisions based on averages. They worry that machines and algorithms are standardized, inflexible, and may miss important evidence. Every user thinks they are a snowflake.

This idea suggests that users are more likely to distrust judgments based on statistics and machine learning because they fear that the model might omit key evidence. Users worry that an AI system is missing important information or factors that could affect its predictions or decisions.

Potential interventions (more in the future):

  • More intuitive design of AI, so that it melts into the background (cases: Netflix, Google Maps).

  • Better communication of the capabilities and usage of AI, especially through trusted sources (e.g. doctors).

  • Personalization to the user. Even adding little things like the user’s name has a huge effect.

  • Letting the user contribute information before the AI starts “working”. Even small changes to the user interfaces have been show to reduce the distriust towards AI. Bring the user into the loop, so that he can show his unique characteristics.

  • Create social and emotional bonds e.g. through warm and nice words.

  • Augmentation instead of automation. Use AI to augment the people in front of the end-user.

Sources

Reply

or to participate.