When the Model Trains You: Induced Belief Revision and Its Implications on Artificial Intelligence Research and Patient Care — A Case Study on Predicting Obstructive Hydronephrosis in Children

Abstract

Exposure to research data and artificial intelligence (AI) model predictions may lead to many sources of bias in clinical decision-making and model evaluation. These include anchoring bias, automation bias, and data leakage. In this case study, we introduce a new source of bias termed "induced belief revision," which we have discovered through our experience developing and testing an AI model to predict obstructive hydronephrosis in children based on their renal ultrasounds. After a silent trial of our hydronephrosis AI model, we observed an unintentional but clinically significant change in practice — characterized by a reduction in nuclear scans from 80 to 58% (P=0.005). This phenomenon occurred in the absence of any identifiable changes in clinical workflow, personnel, practice guidelines, or patient characteristics over time. We postulate that repeated exposures to model predictors and their corresponding labels led to a change in clinical decision-making based on a learned intuition of the model's behavior. There are two types of this phenomenon: data-induced and model-induced belief revision. Data-induced belief revision occurs when clinical end-users develop an unconscious stimulus–response bond based on model features and labels from the data itself. In contrast, model-induced belief revision arises from repeated exposures to model inputs and outputs, resulting in clinicians' anticipating model predictions on unseen data. We hypothesize that model-induced belief revis…