Challenges in the postmarket surveillance of clinical prediction models.

Based on simulations using a previously published model, the bias in observed model performance estimates increases as the model becomes more successful in improving clinical outcomes.

“Based on simulations using a previously published model,4 the resulting bias in observed model performance estimates increases as the model becomes more successful in improving clinical outcomes (Fig. 1).”

• cites [4] Cummings BC, Blackmer JM, Motyka JR, External validation and comparison of a general ward deterioration index between diversely different health systems. Crit Care Med 2023; 51: 775– 786. DOI: 10.1097/CCM.0000000000005837. 36927631 PMC10187626 sourceSupported

Predictive AI models may not generalize across settings or over time due to dataset shift.

“It is well-known that such models may not generalize across settings or over time due to changes in patient populations, practice patterns, and data processes — a phenomenon known as dataset shift.1”

• cites [1] Finlayson SG, Subbaswamy A, Singh K, The clinician and dataset shift in artificial intelligence. N Engl J Med 2021; 385: 283– 286. DOI: 10.1056/NEJMc2104626. 34260843 PMC8665481 sourceSupported

The severity of the CMI problem worsens as clinician trust in the model increases.

“This is consequential because the severity of the problem of CMIs also worsens as clinician trust in the model increases (Fig. 1).”

Including clinician intervention as a predictor can exacerbate the problem if the exchangeability assumption is not satisfied.

“Including intervention as a predictor can exacerbate the problem if the exchangeability assumption is not satisfied.”

• AssumptionThe authors state this as an assumption: The exchangeability assumption for including clinician intervention as a model term requires that subjects receiving and not receiving treatment are conditionally exchangeable given other model covariates, with faithful replication of the data-generating process and no unmeasured confounders.
• AssumptionRests on an assumption: Published prediction models are not typically developed in the same way as causal models for estimating unbiased treatment effects.

Proposed solutions for postdeployment surveillance (withholding outputs, monitoring outcomes as surrogates, including clinician interventions) each have ethical or practical limitations.

“Proposed solutions include withholding model outputs, monitoring outcomes as surrogates, or including clinician interventions in models, each with ethical or practical limitations.”

The lack of effective solutions for postdeployment monitoring can lead to an abundance of models that cannot be later evaluated, tuned, or withdrawn if they become ineffective, leading to patient harm.

“The lack of effective solutions for this problem can lead to an abundance of models that cannot be later evaluated, tuned, or withdrawn if they become ineffective, leading to patient harm.”

Strategies to address more complex scenarios, such as recurrent predictions or multiple interventions, remain unknown.

“Strategies to address more complex scenarios, such as recurrent predictions or multiple interventions, also remain unknown.”

Effective interventions that prevent predicted adverse events alter the rates of the events being predicted.

“If the interventions are effective, adverse events will be prevented, and this, in turn, will alter the rates of the events being predicted.”

If artifactual performance decays are interpreted naively, they may lead to updating or abandoning the model, harming clinical outcomes.

“This can falsely suggest model decay, leading to unwarranted updates or decommissioning, harming clinical outcomes.”

• Also stated in: § Introduction · hedged

  “If interpreted naively, the artifactual performance decays may lead to actions such as updating the model or abandoning the model entirely, both at a detriment to clinical outcomes.”

The CUSUM procedure for monitoring positive and negative predictive values does not apply in more general scenarios, including when a model's goal is to increase the use of treatments already in active use.

“However, it does not apply in more general scenarios, including when a model’s goal is to increase the use of treatments in active use (e.g., allocating existing clinical interventions to patients at high risk of some adverse outcome).”

• cites [7] Feng J, Subbaswamy A, Gossmann A, Designing monitoring strategies for deployed machine learning algorithms: navigating performativity through a causal lens. Proceedings of the Third Conference on Causal Learning and Reasoning. Los Angeles, CA: PMLR, 2024: 587– 608 (https://proceedings.mlr.press/v236/feng24a.html).Not checked yet

  Not checked in the free run.

Causal methods can isolate changes in clinical outcomes directly attributable to a model rather than to secular trends or contemporaneous practice changes.

“Such methods can also isolate changes to clinical outcomes that are directly attributable to a model (rather than to secular trends or contemporaneous practice changes); these effects remain the best gauge of a model’s impact.”

Observed improvements in patient outcomes after model deployment may be due to seasonal or epidemiologic changes or other shifts in clinical practice.

“Observed improvements in patient outcomes may also be due to seasonal or epidemiologic changes or other shifts in clinical practice.”

Confounding medical interventions introduce bias into postdeployment performance assessment, making model predictions appear incorrect.

“A significant issue is postdeployment assessment of predictive AI models due to confounding medical interventions where effective interventions modify outcomes, introducing bias into performance assessment.”

• Also stated in: § Introduction · plain

  “These interventions, also called confounding medical interventions (CMIs), lead to model predictions appearing incorrect on subsequent evaluation.”

• AssumptionRests on an assumption: A common objective of deploying predictive models is to prompt interventions that reduce the risk of adverse outcomes predicted by the model.

Validating models against counterfactual outcomes may enable consistent postdeployment validation despite effective interventions.

“Advanced causal modeling to assess counterfactual outcomes may offer a reliable validation method.”

• Also stated in: § Considerations for Effective Postdeployment Surveillance · hedged

  “If a model’s degraded performance over time is driven by clinicians acting on model predictions to avert adverse outcomes, validating models against counterfactual outcomes — outcomes that would have occurred had a clinician not intervened — may enable consistent postdeployment validation despite effective interventions.”

Once a predictive model is implemented in a clinical workflow, it becomes part of a causal pathway linking its predictions to clinical actions and outcomes.

“The most pertinent among the issues is that once a model is implemented into a clinical workflow, it becomes part of a causal pathway linking the model’s predictions to clinical actions and resulting outcomes.”

Simulation results suggest that the effectiveness of the CUSUM procedure proposed by Feng et al. decreases as clinician trust in model predictions increases.

“However, simulation results suggest that the effectiveness of the CUSUM procedure decreases as clinician trust in the model predictions increases.”

• cites [6] Feng J, Gossmann A, Pennello GA, Petrick N, Sahiner B, Pirracchio R. Monitoring machine learning-based risk prediction algorithms in the presence of performativity. Proceedings of The 27th International Conference on Artificial Intelligence and Statistics. Valencia, Spain: PMLR, 2024: 919– 927 (https://proceedings.mlr.press/v238/feng24b.html).Not checked yet

  Not checked in the free run.