Factors influencing clinician trust in predictive clinical decision support systems for in-hospital deterioration: qualitative descriptive study

Distrust challenges the adoption of machine learning-based CDSSs.

“Such distrust has been shown to challenge the adoption of machine learning–based CDSSs [7-10]”

• cites [7] sourceSupported
• cites [8] sourcePartly supported

  In “Consenting patients” (page not provided), the paper says nurses considered their trust in the deep learning system’s results when deciding whether to follow the study protocol. It does not establish that distrust challenged adoption of machine-learning CDSSs generally; the observed discouragement of participation was attributed primarily to concern about patient hardship. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [9] sourceSupported
• cites [10] sourceNot checked — no full text of the source

It has been difficult for predictive CDSSs to achieve impactful adoption.

“It has been difficult for predictive CDSSs to achieve impactful adoption [10,13,14].”

• cites [10] sourceNot checked — no full text of the source
• cites [13] sourceSupported
• cites [14] sourcePartly supported

  The Discussion reports that the sepsis prediction alert was implemented across a multi-hospital system but had limited impact on clinical practice and outcomes. That supports difficulty achieving impact for this alert, but the paper does not establish that impactful adoption has been difficult for predictive CDSSs generally; it also describes widespread implementation as feasible. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Nurses are target users of predictive CDSSs in the hospital setting.

“However, nurses are also target users of predictive CDSSs in the hospital setting [7,16]”

• cites [7] sourceSupported
• cites [16] sourceNot checked — no full text of the source

CONCERN leverages evidence that nurses alter their documentation behavior or use selected language in their narrative notes when they are concerned about a patient's changing clinical state.

“As such, it leverages evidence that nurses alter their documentation behavior or use selected language in their narrative notes when they are concerned about a patient’s changing clinical state [21-24].”

• cites [21] sourceSupported
• cites [22] sourceSupported
• cites [23] sourceSupported
• cites [24] sourceSupported

Most research on clinician trust has focused on physicians' trust in predictive CDSSs and machine learning-based diagnostic CDSSs.

“Moreover, most research on clinician trust has focused on physicians’ trust in predictive CDSSs and machine learning–based diagnostic CDSSs [2,6,15].”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [6] sourceNot checked — no full text of the source
• cites [15] sourceCould not verify

  The Introduction and Discussion discuss clinicians’ use of ML recommendations and trust, but do not say that most clinician-trust research focuses on physicians or on predictive and ML-based diagnostic CDSSs. The Discussion instead says that few studies evaluate how medical experts change their behavior when interacting with diagnostic recommendations. Searches for prior trust research and diagnostic decision support found no passage establishing the claimed research focus; page numbers were unavailable in our copy. (LLM, conf=0.72)

  Experimental — accuracy not yet measured

Clinicians wanted local explanations for individual patient predictions provided at the point of care.

“Clinicians also wanted explanations for individual patient predictions provided at the point of care.”

Many clinicians indicated an orientation toward rule-based causal logic when describing their desire for local explanations.

“When describing their desire for local explanations, many clinicians indicated an orientation toward rule-based causal logic.”

Some clinicians stated they did not need detailed explanations of CONCERN as it was just one component of their assessments.

“This third subcategory emerged from some clinicians stating that they did not need detailed explanations of CONCERN as it was just 1 component of their assessments:”

For some clinicians, discordance between the CONCERN prediction and their impression of the patient did not diminish their trust because they viewed CONCERN as just one data point that they considered.

“For some clinicians, discordance between the CONCERN prediction and their impression of the patient did not diminish their trust as they viewed CONCERN as just 1 data point that they considered.”

Clinicians primarily judged the accuracy of CONCERN against their own impressions of their patients' risk of deterioration.

“We found that clinicians primarily judged the accuracy of CONCERN against their own impressions of their patients’ risk of deterioration, which may be what clinicians do in real clinical care.”

Reliance on the predictive CDSS in the clinical workflow may influence clinicians' requirements for trust.

“We found that reliance on the predictive CDSS in the clinical workflow may influence clinicians’ requirements for trust.”

Five categories characterized clinicians' perceptions of the accuracy and correctness of CONCERN and predictive CDSSs: concordance, sound methods and data, clinician involvement, systems can only augment, and system strengths and clinician weaknesses.

“Regarding perceived technical competence, five categories characterized clinicians’ perceptions of the accuracy and correctness of CONCERN and predictive CDSSs: (1) concordance, (2) sound methods and data, (3) clinician involvement, (4) systems can only augment, and (5) system strengths and clinician weaknesses.”

Clinicians expressed skepticism of the system's ability to account for unique or complex patient characteristics.

“Clinicians expressed skepticism of the system’s ability to account for unique or complex patient characteristics:”

Clinicians' ability to understand CONCERN was confirmed to be an important factor influencing trust.

“Clinicians’ ability to understand CONCERN was also confirmed to be an important factor influencing trust.”

Clinicians discussed how evidence of CDSSs positively affecting patient care would increase their trust in the system's predictions.

“The evidence theme emerged from clinicians’ discussions of how evidence of CDSSs positively affecting patient care would increase their trust in the system’s predictions.”

Only nurses were concerned about how their documentation affected predictions.

“Although all previous categories were informed by both prescribing providers and nurses, 2 categories were profession-specific: only physicians used analogies, and only nurses were concerned about how their documentation affected predictions.”

Three concepts of the human-computer trust framework (perceived reliability, faith, and personal attachment) were not identified by the participants.

“In all, 3 concepts of the framework (perceived reliability, faith, and personal attachment) were not identified by the participants.”

When CONCERN's predictions aligned with clinicians' impressions of the patient, their trust in the system was positively affected.

“When CONCERN’s predictions aligned with clinicians’ impressions of the patient, their trust in the system was positively affected:”

Evidence describes the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust.

“The second, evidence, described the importance of both macro- (scientific) and micro- (anecdotal) evidence for fostering trust.”

Clinicians wanted global explanations of how the CONCERN model calculates predictions generally.

“Clinicians wanted global explanations, meaning information on how the CONCERN model calculates predictions generally:”

Perceived understandability and perceived technical competence (ie, perceived accuracy) influence clinician trust in predictive CDSSs for in-hospital deterioration.

“Concepts from the human-computer trust conceptual framework—perceived understandability and perceived technical competence (ie, perceived accuracy)—were found to influence clinician trust in predictive CDSSs for in-hospital deterioration.”

• Also stated in: § Inductive Coding · plain

  “The 2 concepts of cognition-based trust, perceived technical competence and perceived understandability, emerged as themes in the inductive coding.”

Clinicians wanted to know the one feature or value that made the patient's prediction yellow or red.

“They wanted to know the one feature or value that made the patient’s prediction yellow or red.”

The study found profession-specific factors characterizing the relationship between understandability and trust.

“Furthermore, we found profession-specific factors characterizing the relationship between understandability and trust.”

Concordance between clinicians' impressions of patients' clinical status and system predictions influences clinicians' perceptions of system accuracy.

“The concordance between clinicians’ impressions of patients’ clinical status and system predictions influenced clinicians’ perceptions of system accuracy.”

• Also stated in: § Inductive Coding · plain

  “Concordance between clinicians’ impression of the patient’s clinical status and CONCERN’s prediction emerged as an important factor influencing clinicians’ perceptions of the accuracy of CONCERN”

Many clinicians described their impression of the quality of data used in the model, data that could improve the model, and the modeling methods to varying levels of granularity.

“When referring to the accuracy of CONCERN, many clinicians described their impression of the quality of data used in the model, data that could improve the model, and the modeling methods to varying levels of granularity.”

Clinicians noted that a CDSS prediction indicating elevated risk had prompted or would prompt them to investigate further, either via EHR data review or discussion with another team member.

“Clinicians also noted that a CDSS prediction indicating elevated risk had prompted or would prompt them to investigate further, either via EHR data review or discussion with another team member:”

Some clinicians discussed how CDSSs provide reason for them to further examine a patient's clinical status.

“Others discussed how CDSSs provide reason for them to further examine a patient’s clinical status.”

Clinicians described predictive CDSSs as better equipped to make predictions in situations where the change to the patient's state is gradual rather than rapid.

“Clinicians described predictive CDSSs as better equipped to make predictions in situations where the change to the patient’s state is gradual rather than rapid:”

Clinicians described the importance of anecdotal reports of positive impact.

“Clinicians also described the importance of anecdotal reports of positive impact:”

A lack of concordance between CONCERN's predictions and clinicians' assessments decreased their trust.

“Conversely, clinicians expressed that a lack of concordance between CONCERN’s predictions and their assessments decreased their trust:”

Perceived understandability was characterized by four categories: explanations, understanding must be acquired, the equivalent of... (physicians only), and Am I doing this right? (nurses only).

“Perceived understandability was characterized by four categories: (1) explanations, (2) understanding must be acquired, (3) the equivalent of... (physicians only), and (4) Am I doing this right? (nurses only).”

The study identified additional factors influencing clinician trust: evidence, perceived actionability, and equitability.

“We confirmed that perceived understandability and perceived technical competence influence clinicians’ trust in predictive CDSSs as well as identified additional factors: evidence, perceived actionability, and equitability.”

New themes emerged from the data analysis that do not map onto the conceptual framework: evidence, perceived actionability, and equitability.

“Finally, new themes emerged from the data analysis that do not map onto the conceptual framework. These included (1) evidence, (2) perceived actionability, and (3) equitability.”

Some nurses said they had or would change their documentation behavior in attempts to make the prediction reflect their impression of the patient.

“Some said that they had or would change their documentation behavior in attempts to make the prediction reflect their impression of the patient:”

Emergent codes between sites A and B did not differ significantly.

“Emergent codes between sites A and B did not differ significantly (Table S2 in Multimedia Appendix 1).”

Three additional themes emerged from the inductive analysis: perceived actionability, evidence, and equitability.

“In total, 3 additional themes emerged from the inductive analysis.”

• Also stated in: § Inductive Coding · plain

  “In addition, three new themes reflecting clinicians’ trust in predictive CDSSs were identified: (1) evidence, (2) perceived actionability, and (3) equitability (Figure 3).”

Understandability is influenced by system explanations, both global and local, as well as training.

“Understandability was influenced by system explanations, both global and local, as well as training.”

Only physicians used analogies when describing their understanding of CONCERN.

“Although all previous categories were informed by both prescribing providers and nurses, 2 categories were profession-specific: only physicians used analogies, and only nurses were concerned about how their documentation affected predictions.”

Clinicians described their trust as being influenced by their perceptions of the accuracy and correctness of CONCERN.

“Clinicians described their trust as being influenced by their perceptions of the accuracy and correctness of CONCERN.”

The extent to which clinicians rely on the predictive CDSS was shown to influence both the impact of discordant predictions and the detail needed from explanations.

“Furthermore, the extent to which clinicians rely on the predictive CDSS was shown to influence both the impact of discordant predictions and the detail needed from explanations.”

Some clinicians described themselves as being self-motivated to learn about CONCERN.

“In this subcategory, some clinicians described themselves as being self-motivated to learn.”

Clinicians frequently described scenarios in which data that the system uses would be missing, including emergencies, rapid deterioration, new patients, or when clinicians are burdened by work or documentation.

“Clinicians frequently described scenarios in which data that the system uses would be missing. Examples included emergencies, rapid deterioration, new patients, or when clinicians are burdened by work or documentation:”

When prompted, clinicians said that an accuracy metric would be helpful.

“When prompted, they said that an accuracy metric would be helpful, but differences may be attributable to context.”

The findings were largely similar between nurses and prescribing providers.

“The findings were largely similar between nurses and prescribing providers.”

Many clinicians mentioned they would probably rely more on predictive CDSSs during the night shift when they are assigned to care for more patients.

“For example, many clinicians mentioned that they would probably rely more on predictive CDSSs during the night shift, when they are assigned to care for more patients:”

No clinician requested a certainty score or any type of accuracy metric in the interviews.

“Tonekaboni et al [2] reported that clinicians would like to see a certainty score presented with the CDSS prediction; however, no clinician requested this or any type of accuracy metric in our interviews.”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The study found that perceptions of trust were largely the same between nurses and prescribing providers.

“We found that perceptions of trust were largely the same between nurses and prescribing providers.”

When the logic behind predictions was not understood, physicians did not trust them.

“When the logic behind the predictions was not understood, the physicians did not trust them.”

• cites [6] sourceNot checked — no full text of the source

Clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy.

“Evidence shows that clinician trust in predictive CDSSs is influenced by perceived understandability and perceived accuracy.”

• Also stated in: § Objectives · boosted

  “Our investigation is guided by a conceptual framework, the human-computer trust framework [17], which accounts for the aforementioned known factors that influence trust—perceived understandability and accuracy.”

• cites [17] MadsenMGregorSMeasuring human-computer trustProceedings of the 11 th Australasian Conference on Information Systems2000Proceedings of the 11 th Australasian Conference on Information Systems2000Brisbane Qld AustraliaNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Presenting clinicians with a model's overall accuracy is not sufficient for establishing trust in predictive CDSSs.

“Research indicates that presenting clinicians with a model’s overall accuracy is not sufficient for establishing trust in predictive CDSSs [2].”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Understandability and explainability are positively associated with physicians' trust in a machine learning-based CDSS.

“In a study of physicians’ ability to understand and explain a machine learning–based CDSS’s logic, Diprose et al [6] found that both understandability and explainability were positively associated with trust.”

• cites [6] sourceNot checked — no full text of the source

The logic behind predictions in machine learning-based CDSSs can be difficult or impossible to make transparent to clinical end users.

“A challenge is that the logic behind predictions in machine learning–based CDSSs can be difficult or impossible to make transparent to clinical end users.”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [3] sourceNot checked — no full text of the source
• cites [4] Holzinger, A, 2017, arXivNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

When some clinicians were asked how they would determine that the CONCERN tool was accurate, they answered by expressing a desire to understand it more thoroughly, indicating a primacy of understandability over accuracy.

“In fact, when some clinicians were asked how they would determine that the CONCERN tool was accurate, they answered by expressing a desire to understand it more thoroughly—indicating a primacy of understandability over accuracy, as others have found [2].”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The 2 deductive coders achieved an overall Cohen κ of 0.81.

“The 2 deductive coders achieved an overall Cohen κ of 0.81.”

Prior studies on explainability and trust in predictive CDSSs were conducted using simulated rather than live implemented predictive CDSSs.

“The latter two were conducted by referring to simulated as opposed to live implemented predictive CDSSs.”

• cites [2] TonekaboniSJoshiSMcCraddenMGoldenbergAWhat clinicians want: contextualizing explainable machine learning for clinical end useProceedings of Machine Learning Research2019Machine Learning for HealthcareAug 8-10, 2019Ann Arbor, MINot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [6] sourceNot checked — no full text of the source

Data saturation occurred at the 11th interview.

“Data saturation occurred at the 11th interview (Table S2 in Multimedia Appendix 1).”

The specific questions about CONCERN and about predictive CDSSs generally, as well as heterogeneity in exposure to CONCERN, limit the ability to know which findings are unique to CONCERN.

“Our specific questions about CONCERN (and about predictive CDSSs generally) as well as the heterogeneity in exposure to CONCERN limit our ability to know which findings are unique to CONCERN.”