To Recommend or Not to Recommend: Designing and Evaluating AI-Enabled Decision Support for Time-Critical Medical Events

Classified as probably not a substantial claim.

Information-driven support may provide users with more agency and autonomy than recommendation-driven support.

“In addition to supporting decision making, information-driven support may provide users with more agency and autonomy than recommendation-driven support [82, 83].”

• cites [82] Zhang Shao, Yu Jianing, Xu Xuhai, Yin Changchang, Lu Yuxuan, Yao Bingsheng, Tory Melanie, Padilla Lace M, Caterino Jeffrey, Zhang Ping, 2024. Rethinking human-ai collaboration in complex medical decision making: A case study in sepsis diagnosis. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1– 18. PMC11149368 38835626 sourcePartly supported

  Section 3.2.5 (page unavailable) says AI output that suggests a final diagnosis can challenge clinicians’ authority. Section 6.1 (page unavailable) argues that information and support at intermediate stages can preserve clinicians’ role as final decision-makers. But the proposed system also gives laboratory-test recommendations, so the paper does not support a general contrast between information-driven and recommendation-driven support. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [83] Zhang Zelun Tony, Feger Sebastian S, Dullenkopf Lucas, Liao Rulu, Süsslin Lukas, Liu Yuanting, and Butz Andreas. 2024. Beyond Recommendations: From Backward to Forward AI Support of Pilots’ Decision-Making Process. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1– 32.39286336Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim.

Some intensive care and emergency medicine physicians preferred receiving information synthesis from AI-enabled systems to help them generate and test hypotheses instead of risk prediction scores that could challenge their authority.

“In one study, some intensive care and emergency medicine physicians preferred receiving information synthesis from AI-enabled systems to help them generate and test hypotheses instead of risk prediction scores that could challenge their authority [82].”

• cites [82] Zhang Shao, Yu Jianing, Xu Xuhai, Yin Changchang, Lu Yuxuan, Yao Bingsheng, Tory Melanie, Padilla Lace M, Caterino Jeffrey, Zhang Ping, 2024. Rethinking human-ai collaboration in complex medical decision making: A case study in sepsis diagnosis. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1– 18. PMC11149368 38835626 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Radiologists in one study found it challenging to determine the correctness of AI predictions, leading to poor decisions when inaccurate predictions were provided.

“For example, radiologists in one study found it challenging to determine the correctness of AI predictions, leading to poor decisions when inaccurate predictions were provided [78].”

• cites [78] Yu Feiyang, Moehring Alex, Banerjee Oishi, Salz Tobias, Agarwal Nikhil, and Rajpurkar Pranav. 2024. Heterogeneity and predictors of the effects of AI assistance on radiologists. Nature Medicine (2024), 1– 13. PMC10957478 38504016 sourcePartly supported

  The Discussion reports that large AI prediction errors tended to harm radiologists’ performance and suggests that radiologists struggled to distinguish accurate from inaccurate predictions. The claim presents that difficulty as an established finding, whereas the paper cautions against inferring the underlying cognitive mechanism from these results. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Providers' interpretation of system output is more important than the output itself because providers are responsible for determining treatment plans.

“Because providers are responsible for determining treatment plans, their interpretation of system output is more important than the output itself [69].”

• cites [69] Vasey Baptiste, Ursprung Stephan, Beddoe Benjamin, Taylor Elliott H, Marlow Neale, Bilbro Nicole, Watkinson Peter, and McCulloch Peter. 2021. Association of clinician diagnostic performance with machine learning–based decision support systems: a systematic review. JAMA network open 4, 3 (2021), e211276– e211276. 33704476 PMC7953308 sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Providers rated the quality of advice from other providers more poorly when it was labeled as AI generated.

“This finding aligns with prior work, which found that providers rated the quality of advice from other providers more poorly when it was labeled as AI generated [28].”

• cites [28] Gaube Susanne, Suresh Harini, Raue Martina, Merritt Alexander, Berkowitz Seth J, Lermer Eva, Coughlin Joseph F, Guttag John V, Colak Errol, and Ghassemi Marzyeh. 2021. Do as AI say: susceptibility in deployment of clinical decision-aids. NPJ digital medicine 4, 1 (2021), 31. PMC7896064 33608629 sourcePartly supported

  The Results section, “Advice quality ratings display algorithmic aversion in experts,” reports that radiologists rated human-generated advice more poorly when it was presented as AI advice. The finding did not extend to the less task-expert physicians, so the unqualified reference to “providers” is broader than the source supports. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

May combine two claims.

The simulated system had an accuracy of 75%, a level used in prior studies evaluating AI support strategies.

“As a result, our simulated system had an accuracy of 75%, a level used in prior studies evaluating the effects of different AI support strategies on decision making [8, 9].”

• cites [8] Buçinca Zana, Malaya Maja Barbara, and Gajos Krzysztof Z. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1– 21.36644216Not checked yet

  Not checked in the free run.

• cites [9] Buçinca Zana, Swaroop Siddharth, Paluch Amanda E, Murphy Susan A, and Gajos Krzysztof Z. 2024. Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning. arXiv preprint arXiv:2403.05911 (2024). sourceSupported

May combine two claims.

Patients are often transfused unnecessarily or die from hemorrhage before receiving blood.

“Because patients are often transfused unnecessarily or die from hemorrhage before receiving blood [33], we adjudicated the need for blood transfusion in these potential cases.”

• cites [33] Hess John R and Hiippala Seppo. 2005. Optimizing the use of blood products in trauma care. Critical care 9, 5 (2005), 1– 5. PMC3226118 16221314 sourcePartly supported

  The Abstract says not everyone who receives blood benefits, and “Patterns of blood use” describes patients receiving blood despite no benefit. The Introduction identifies hemorrhage as a major cause of death, while “Can we use less blood?” says severely injured patients can die quickly without resuscitation. The paper does not establish that patients often die from hemorrhage specifically before receiving blood, or quantify how often unnecessary transfusion occurs. (LLM, conf=0.83)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

AI systems may need to support multiple human-centric objectives, such as learning, collaboration, and agency.

“In addition to improving decision making, AI systems may need to support multiple human-centric objectives, such as learning, collaboration, and agency [9].”

• cites [9] Buçinca Zana, Swaroop Siddharth, Paluch Amanda E, Murphy Susan A, and Gajos Krzysztof Z. 2024. Towards Optimizing Human-Centric Objectives in AI-Assisted Decision-Making With Offline Reinforcement Learning. arXiv preprint arXiv:2403.05911 (2024). sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing).

Cognitive forcing strategies, such as diagnostic timeouts, may be effective in preventing failed heuristics by prompting providers to analytically engage with their decisions.

“Cognitive forcing strategies, such as diagnostic timeouts, may be effective in preventing failed heuristics by prompting providers to analytically engage with their decisions [19, 40].”

• cites [19] Croskerry Pat, Singhal Geeta, and Mamede Sílvia. 2013. Cognitive debiasing 2: impediments to and strategies for change. BMJ quality & safety 22, Suppl 2 (2013), ii65– ii72. PMC3786644 23996094 sourcePartly supported

  In “Strategies for cognitive debiasing” (Tables 1–2; page unavailable), the paper says cognitive forcing strategies can mitigate biases leading to decision errors and describes deliberate reflection as a way to check initial diagnoses. It mentions a planned time-out in the operating room, but does not identify diagnostic timeouts as a cognitive forcing strategy or establish their effectiveness in preventing failed heuristics. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [40] Lambe Kathryn Ann, O’Reilly Gary, Kelly Brendan D, and Curristan Sarah. 2016. Dual-process cognitive interventions to enhance diagnostic reasoning: a systematic review. BMJ quality & safety 25, 10 (2016), 808– 820. 26873253 sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

AI-enabled decision-support systems have often been designed for clinical settings that rely on image interpretation, such as radiology and pathology.

“AI-enabled decision-support systems have often been designed for clinical settings that rely on image interpretation, such as radiology and pathology [12, 16, 25, 31, 65, 77].”

• cites [12] Cai Carrie J, Reif Emily, Hegde Narayan, Hipp Jason, Kim Been, Smilkov Daniel, Wattenberg Martin, Viegas Fernanda, Corrado Greg S, Stumpe Martin C, 2019. Human-centered tools for coping with imperfect algorithms during medical decision-making. In Proceedings of the 2019 chi conference on human factors in computing systems. 1– 14.Not checked yet

  Not checked in the free run.

• cites [16] Corti Lorenzo, Oltmans Rembrandt, Jung Jiwon, Balayn Agathe, Wijsenbeek Marlies, and Yang Jie. 2024. “ It Is a Moving Process”: Understanding the Evolution of Explainability Needs of Clinicians in Pulmonary Medicine. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1– 21.Not checked yet

  Not checked in the free run.

• cites [25] Fogliato Riccardo, Chappidi Shreya, Lungren Matthew, Fisher Paul, Wilson Diane, Fitzke Michael, Parkinson Mark, Horvitz Eric, Inkpen Kori, and Nushi Besmira. 2022. Who goes first? Influences of human-AI workflow on decision making in clinical imaging. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1362– 1374.Not checked yet

  Not checked in the free run.

• cites [31] Gu Hongyan, Yang Chunxu, Haeri Mohammad, Wang Jing, Tang Shirley, Yan Wenzhong, He Shujin, Williams Christopher Kazu, Magaki Shino, and Chen ang’Anthony’. 2023. Augmenting Pathologists with NaviPath: Design and Evaluation of a Human-AI Collaborative Navigation System. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1– 19.Not checked yet

  Not checked in the free run.

• cites [65] Thieme Anja, Rajamohan Abhijith, Cooper Benjamin, Groombridge Heather, Simister Robert, Wong Barney, Woznitza Nicholas, Pinnock Mark Ames, Wetscherek Maria Teodora, Morrison Cecily, 2024. Challenges for Responsible AI Design and Workflow Integration in Healthcare: A Case Study of Automatic Feeding Tube Qualification in Radiology. arXiv preprint arXiv:2405.05299 (2024). sourceSupported
• cites [77] Yildirim Nur, Richardson Hannah, Maria Teodora Wetscherek Junaid Bajwa, Jacob Joseph, Pinnock Mark Ames, Harris Stephen, De Castro Daniel Coelho, Bannur Shruthi, Hyland Stephanie, 2024. Multimodal healthcare AI: identifying and designing clinically relevant vision-language applications for radiology. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1– 22.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based). May combine two claims.

Some providers appreciated the ability to use the recommendations to check their decisions, while others thought the recommendations lacked nuance and could bias their decision making.

“Some providers appreciated the ability to use the recommendations to check their decisions, while others thought the recommendations lacked nuance and could bias their decision making.”

• AssumptionRests on an assumption: A simulated AI system can validly generate and control recommendations and information synthesis for evaluating human-AI interaction strategies.

May combine two claims.

Thirty-five physicians and advanced practice providers completed the experiment and passed the attention checks.

“Thirty-five physicians and advanced practice providers completed the experiment and passed the attention checks (Table 1, full demographic information in Appendix A).”

May combine two claims.

Participants ranked AI Information Synthesis (z = 2.39, p = 0.05) and AI Information and Recommendations (z = 2.57, p = 0.03) as more effective at supporting decision making than No AI Support.

“In the post-hoc pairwise comparisons for effectiveness, participants ranked AI Information Synthesis (z = 2.39, p=0.05) and AI Information and Recommendations (z = 2.57, p=0.03) as more effective at supporting decision making than No AI Support.”

May combine two claims.

The study collected 816 decision-making instances: 408 about blood transfusion and 408 about neurosurgical intervention.

“We collected a total of 816 decision-making instances: 408 about blood transfusion and 408 about neurosurgical intervention.”

May combine two claims.

In the AI Information and Recommendations condition, final answers selected during the observation period were accurate 64% of the time and during the decision period 68% of the time.

“When AI Information and Recommendations were provided, final answers selected during the observation period were accurate 64% of the time and final answers selected during the decision period were accurate 68% of the time.”

May combine two claims.

There was no difference in diagnostic accuracy between AI Information Synthesis and No AI Support (z = 0.98, p = 0.33) or between AI Information and Recommendations and AI Information Synthesis (z = 1.78, p = 0.15).

“We did not find a difference in diagnostic accuracy between AI Information Synthesis and No AI Support (z = 0.98, p=0.33) or AI Information and Recommendations and AI Information Synthesis (z = 1.78, p=0.15).”

• Also stated in: § DISCUSSION · plain

  “In contrast, we did not find a significant difference in diagnostic accuracy between only AI information synthesis and no AI support.”

May combine two claims.

Participants received correct recommendations but made incorrect decisions.

“Although providing AI information and recommendations improved diagnostic accuracy, we still observed instances where participants received correct recommendations but made incorrect decisions.”

May combine two claims.

In borderline risk instances, the type of AI support had a significant association with diagnostic accuracy (χ2(2, N = 510) = 8.28, p = 0.02), and decisions were more likely correct with AI Information and Recommendations than No AI Support (OR = 2.23, 95% CI [1.29, 3.87], z = 2.85, p = 0.01).

“The type of AI support again had a significant association with diagnostic accuracy (χ2(2, N = 510) = 8.28, p = 0.02). Decisions were more likely to be correct when AI Information and Recommendations were provided as compared to No AI Support (OR=2.23, 95% CI [1.29, 3.87], z = 2.85, p=0.01).”

May combine two claims.

Eighteen participants indicated that they had considered recommendations from the system during decision making, noting that recommendations often arrived after they had made an initial decision.

“Eighteen participants indicated that they had considered recommendations from the system during decision making, noting that recommendations often arrived after they had made an initial decision.”

May combine two claims.

Participants described using the AI information synthesis to review findings from patient examinations, identify abnormalities, and check their logic.

“Participants described using the AI information synthesis to review findings from patient examinations, identify abnormalities, and check their logic.”

May combine two claims.

There were differences in participant rankings of the three AI support types for effectiveness (χ2(2, N = 35) = 11.20, p < 0.01), efficiency (χ2(2, N = 35) = 6.70, p = 0.04), and overall preference (χ2(2, N = 35) = 8.91, p = 0.01).

“We found differences in participant rankings of the three AI support types for effectiveness (χ2(2, N = 35) = 11.20, p < 0.01), efficiency (χ2(2, N = 35) = 6.70, p = 0.04), and overall preference (χ2(2, N = 35) = 8.91, p = 0.01) (Figure 9).”

May combine two claims.

Participants changed their initial answers in 89 instances (11%), more frequently for neurosurgical intervention (n = 57) than for blood transfusion (n = 32).

“Participants changed their initial answers in 89 instances (11%), sometimes changing an answer to the same question multiple times. Answers were more frequently changed for decisions about neurosurgical intervention (n=57) than for decisions about blood transfusion (n=32).”

The quote only partly states this claim.

Overall, participants had fewer concerns about receiving AI information synthesis during real practice than about recommendations.

“Overall, participants had fewer concerns about receiving AI information synthesis during real practice.”

May combine two claims.

Providers in trauma resuscitation had polarizing perceptions and uses of AI recommendations, with some participants raising concerns that AI recommendations could lead to conflict and distrust among team members.

“Similarly, we found that providers in trauma resuscitation had polarizing perceptions and uses of AI recommendations, with some participants raising concerns that AI recommendations could lead to conflict and distrust among team members.”

Classified as probably not a substantial claim.

Other participants expressed concerns about their agency to ignore recommendations.

“Others expressed concerns about their agency to ignore recommendations.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants in the design research phase highlighted that decision-support displays should be viewed by the entire team to support a shared mental model.

“Participants in our design research phase also highlighted that decision-support displays should be viewed by the entire team to support a shared mental model.”

May combine two claims.

Some providers had negative perceptions of the recommendations, and were concerned that the recommendations could be incorrect or lead to biased decision making.

“However, some providers had negative perceptions of the recommendations, and were concerned that the recommendations could be incorrect or lead to biased decision making.”

May combine two claims.

Recommendations usually appeared after initial decisions, with some participants describing how the recommendations prompted them to check their decisions.

“As a result, recommendations usually appeared after initial decisions, with some participants describing how the recommendations prompted them to check their decisions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

One provider described how they may assess the trade-offs of waiting for a CT scan to perform neurosurgical intervention.

“For example, one provider described how they may assess the trade-offs of waiting for a CT scan to perform neurosurgical intervention.”

May combine two claims.

Few participants had concerns about receiving information synthesis, which was considered objective.

“However, few participants had concerns about receiving information synthesis, which was considered objective.”

May combine two claims.

Participants often raised concerns about over-relying on AI recommendations and making biased or incorrect decisions.

“Participants often raised concerns about over-relying on AI recommendations and making biased or incorrect decisions.”

Probably not a substantial claim: may not pass S4 (based).

Controlling the points when recommendations are provided in the clinical workflow during time-critical medical events is more challenging due to the dynamic and uncertain nature of patient information.

“Controlling the points when recommendations are provided in the clinical workflow during time-critical medical events is more challenging due to the dynamic and uncertain nature of patient information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Providers may have concerns that clinical decision-support systems will infringe on their agency.

“Finally, providers may have concerns that these systems will infringe on their agency [5, 66, 70].”

• cites [5] Berge Arngeir, Guribye Frode, Fotland Siri-Linn Schmidt, Fonnes Gro, Johansen Ingrid H, and Trattner Christoph. 2023. Designing for Control in Nurse-AI Collaboration During Emergency Medical Calls. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 1339– 1352.Not checked yet

  Not checked in the free run.

• cites [66] Valenta Annette L, Browning Margaret M, Weddle Timothy E, Stevenson Greer WP, Boyd Andrew D, and Hynes Denise M. 2010. Physician perceptions of clinical reminders. In Proceedings of the 1st ACM International Health Informatics Symposium. 710– 717.Not checked yet

  Not checked in the free run.

• cites [70] Wang Dakuo, Wang Liuping, Zhang Zhan, Wang Ding, Zhu Haiyi, Gao Yvonne, Fan Xiangmin, and Tian Feng. 2021. “Brilliant AI doctor” in rural clinics: Challenges in AI-powered clinical decision support system deployment. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1– 18.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Compared to no AI support, only providing AI explanations and omitting recommendations improved learning gains of crowdsourcing platform users asked to make nutrition-related decisions.

“For example, compared to no AI support, only providing AI explanations and omitting recommendations improved learning gains of crowdsourcing platform users asked to make nutrition-related decisions [27].”

• cites [27] Gajos Krzysztof Z and Mamykina Lena. 2022. Do people engage cognitively with AI? Impact of AI assistance on incidental learning. In 27th international conference on intelligent user interfaces. 794– 806.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Providers may have concerns that clinical decision-support systems will require additional cognitive effort in already time- and resource-constrained scenarios.

“They may also have concerns that these systems will require additional cognitive effort in already time- and resource-constrained scenarios [2, 35, 36, 81].”

• cites [2] Bach Anne Kathrine Petersen, Nørgaard Trine Munch, Brok Jens Christian, and van Berkel Niels. 2023. “ If I Had All the Time in the World”: Ophthalmologists’ Perceptions of Anchoring Bias Mitigation in Clinical AI Support. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1– 14.Not checked yet

  Not checked in the free run.

• cites [35] Jacobs Maia, He Jeffrey, Pradier Melanie F., Lam Barbara, Ahn Andrew C, McCoy Thomas H, Perlis Roy H, Doshi-Velez Finale, and Gajos Krzysztof Z. 2021. Designing AI for trust and collaboration in time-constrained medical decisions: a sociotechnical lens. In Proceedings of the 2021 chi conference on Human Factors in computing systems. 1– 14.Not checked yet

  Not checked in the free run.

• cites [36] Kaltenhauser Annika, Rheinstädter Verena, Butz Andreas, and Wallach Dieter P. 2020. “ You Have to Piece the Puzzle Together” Implications for Designing Decision Support in Intensive Care. In Proceedings of the 2020 ACM Designing Interactive Systems Conference. 1509– 1522.Not checked yet

  Not checked in the free run.

• cites [81] Zhang Minfan, Ehrmann Daniel, Mazwi Mjaye, Eytan Danny, Ghassemi Marzyeh, and Chevalier Fanny. 2022. Get To The Point! Problem-Based Curated Data Views To Augment Care For Critically Ill Patients. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1– 13.Not checked yet

  Not checked in the free run.

May combine two claims.

When providers cannot accurately assess the quality of AI output, they may ignore correct advice from the system or follow incorrect advice.

“When providers cannot accurately assess the quality of the output, they may ignore correct advice from the system or follow incorrect advice.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cognitive forcing strategies that delay showing AI recommendations or require a provisional decision can reduce acceptance of AI recommendations regardless of their accuracy.

“For example, some cognitive forcing strategies intentionally delay showing users the AI recommendations or require making a provisional decision before viewing the recommendation. This approach can reduce acceptance of AI recommendations regardless of their accuracy [8, 25, 52].”

• cites [8] Buçinca Zana, Malaya Maja Barbara, and Gajos Krzysztof Z. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1– 21.36644216Not checked yet

  Not checked in the free run.

• cites [25] Fogliato Riccardo, Chappidi Shreya, Lungren Matthew, Fisher Paul, Wilson Diane, Fitzke Michael, Parkinson Mark, Horvitz Eric, Inkpen Kori, and Nushi Besmira. 2022. Who goes first? Influences of human-AI workflow on decision making in clinical imaging. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1362– 1374.Not checked yet

  Not checked in the free run.

• cites [52] Park Joon Sung, Barber Rick, Kirlik Alex, and Karahalios Karrie. 2019. A slow algorithm improves users’ assessments of the algorithm’s accuracy. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1– 15.34322658Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

In some studies, providing explanations with recommendations reduced over-reliance on incorrect predictions or recommendations.

“In some studies, providing explanations with recommendations reduced over-reliance on incorrect predictions or recommendations [41, 68].”

• cites [41] Lee Min Hun and Chew Chong Jun. 2023. Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1– 22.Not checked yet

  Not checked in the free run.

• cites [68] Vasconcelos Helena, Jörke Matthew, Grunde-McLaughlin Madeleine, Gerstenberg Tobias, Bernstein Michael S, and Krishna Ranjay. 2023. Explanations can reduce overreliance on ai systems during decision-making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1– 38.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Users in clinical and non-clinical settings perceived systems with cognitive forcing strategies as less useful than other AI systems.

“Users in clinical and non-clinical settings perceived systems with these cognitive forcing strategies as less useful than other AI systems [8, 25].”

• cites [8] Buçinca Zana, Malaya Maja Barbara, and Gajos Krzysztof Z. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1– 21.36644216Not checked yet

  Not checked in the free run.

• cites [25] Fogliato Riccardo, Chappidi Shreya, Lungren Matthew, Fisher Paul, Wilson Diane, Fitzke Michael, Parkinson Mark, Horvitz Eric, Inkpen Kori, and Nushi Besmira. 2022. Who goes first? Influences of human-AI workflow on decision making in clinical imaging. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 1362– 1374.Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

When making decisions under time pressure, people may be more likely to use recognition of past experiences and heuristics instead of more analytical strategies.

“When making decisions under time pressure, people may be more likely to use recognition of past experiences and heuristics instead of more analytical strategies, such as systematically evaluating different options [18, 38, 54].”

• cites [18] Croskerry Pat. 2003. The importance of cognitive errors in diagnosis and strategies to minimize them. Academic medicine 78, 8 (2003), 775– 780. 12915363 sourceNot checked — no full text of the source
• cites [38] Klein Gary A. 1993. A recognition-primed decision (RPD) model of rapid decision making. Decision making in action: Models and methods 5, 4 (1993), 138– 147.Not checked yet

  Not checked in the free run.

• cites [54] Rieskamp Jörg and Hoffrage Ulrich. 1999. When do people use simple heuristics, and how can we tell? (1999).Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Blood transfusion and neurosurgical intervention are administered when treating two of the most frequent causes of preventable deaths in pediatric trauma resuscitation, severe blood loss and traumatic brain injury.

“These LSIs are administered when treating two of the most frequent causes of preventable deaths in pediatric trauma resuscitation, severe blood loss and traumatic brain injury [23].”

• cites [23] Drake Stacy A, Holcomb John B, Yang Yijiong, Thetford Caitlin, Myers Lauren, Brock Morgan, Wolf Dwayne A, Cron Stanley, Persse David, McCarthy James, 2020. Establishing a regional trauma preventable/potentially preventable death rate. Annals of surgery 271, 2 (2020), 375– 382. 30067544 sourceNot checked — no full text of the source

May combine two claims.

There are two socio-technical barriers to implementing AI recommendations in time-critical medical events: determining when to present AI recommendations and supporting multiple human-centric objectives in addition to improving decision making.

“We highlight two socio-technical barriers to implementing AI recommendations in time-critical medical events: (1) determining when to present AI recommendations and (2) supporting multiple human-centric objectives in addition to improving decision making.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Providing recommendations after a user had made a decision led to no learning gains.

“In contrast, providing recommendations after a user had made a decision led to no learning gains.”

• cites [27] Gajos Krzysztof Z and Mamykina Lena. 2022. Do people engage cognitively with AI? Impact of AI assistance on incidental learning. In 27th international conference on intelligent user interfaces. 794– 806.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing).

Providing explanations with recommendations has shown mixed results.

“Although explainable AI techniques can mitigate common biases in decision making [71], providing explanations with recommendations has shown mixed results [3, 8, 11, 41, 51, 68].”

• cites [3] Bansal Gagan, Wu Tongshuang, Zhou Joyce, Fok Raymond, Nushi Besmira, Kamar Ece, Ribeiro Marco Tulio, and Weld Daniel. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1– 16.Not checked yet

  Not checked in the free run.

• cites [8] Buçinca Zana, Malaya Maja Barbara, and Gajos Krzysztof Z. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1– 21.36644216Not checked yet

  Not checked in the free run.

• cites [11] Bussone Adrian, Stumpf Simone, and O’Sullivan Dympna. 2015. The role of explanations on trust and reliance in clinical decision support systems. In 2015 international conference on healthcare informatics. IEEE, 160– 169.Not checked yet

  Not checked in the free run.

• cites [41] Lee Min Hun and Chew Chong Jun. 2023. Understanding the Effect of Counterfactual Explanations on Trust and Reliance on AI for Human-AI Collaborative Clinical Decision Making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1– 22.Not checked yet

  Not checked in the free run.

• cites [51] Panigutti Cecilia, Beretta Andrea, Giannotti Fosca, and Pedreschi Dino. 2022. Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1– 9.Not checked yet

  Not checked in the free run.

• cites [68] Vasconcelos Helena, Jörke Matthew, Grunde-McLaughlin Madeleine, Gerstenberg Tobias, Bernstein Michael S, and Krishna Ranjay. 2023. Explanations can reduce overreliance on ai systems during decision-making. Proceedings of the ACM on Human-Computer Interaction 7, CSCW1 (2023), 1– 38.Not checked yet

  Not checked in the free run.

May combine two claims.

Both AI information synthesis and AI information and recommendations may have functioned as process-oriented support, with patient information synthesized on the display as it was presented in the vignette.

“Findings from the experiment suggest that both strategies may have functioned as a type of process-oriented support [83], with patient information synthesized on the display as it was being presented in the vignette.”

• cites [83] Zhang Zelun Tony, Feger Sebastian S, Dullenkopf Lucas, Liao Rulu, Süsslin Lukas, Liu Yuanting, and Butz Andreas. 2024. Beyond Recommendations: From Backward to Forward AI Support of Pilots’ Decision-Making Process. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1– 32.39286336Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: A simulated AI system can validly generate and control recommendations and information synthesis for evaluating human-AI interaction strategies.

Classified as probably not a substantial claim. May combine two claims.

Although recommendations may improve decision making, recommendations may also negatively impact provider collaboration and agency.

“Although recommendations may improve decision making, recommendations may also negatively impact provider collaboration and agency.”

Probably not a substantial claim: may not pass S2 (checkable).

These results add new knowledge to the limited evidence that human-AI collaboration can enhance decision making of domain experts in time- and resource-constrained settings.

“These results add new knowledge to the limited evidence that human-AI collaboration can enhance decision making of domain experts in time- and resource-constrained settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

In other studies, over-reliance on system outputs increased with explanations.

“In others, over-reliance on system outputs increased with explanations [3].”

• cites [3] Bansal Gagan, Wu Tongshuang, Zhou Joyce, Fok Raymond, Nushi Besmira, Kamar Ece, Ribeiro Marco Tulio, and Weld Daniel. 2021. Does the whole exceed its parts? the effect of ai explanations on complementary team performance. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1– 16.Not checked yet

  Not checked in the free run.

May combine two claims.

Recommendations appeared at the end of the vignette and often served as a cognitive forcing function.

“Because recommendations appeared at the end of the vignette, they often served as a cognitive forcing function [8].”

• cites [8] Buçinca Zana, Malaya Maja Barbara, and Gajos Krzysztof Z. 2021. To trust or to think: cognitive forcing functions can reduce overreliance on AI in AI-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1– 21.36644216Not checked yet

  Not checked in the free run.

• AssumptionRests on an assumption: A simulated AI system can validly generate and control recommendations and information synthesis for evaluating human-AI interaction strategies.

Classified as probably not a substantial claim. May combine two claims.

To mitigate learning and order effects, participants were randomly split into groups, with the order of conditions counterbalanced across the groups.

“To mitigate learning and order effects, we randomly split participants into groups, with the order of conditions counterbalanced across the groups.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Information-driven strategies do not center on providing recommendations.

“Unlike recommendation-driven strategies, information-driven strategies do not center on providing recommendations [27, 45, 83].”

• cites [27] Gajos Krzysztof Z and Mamykina Lena. 2022. Do people engage cognitively with AI? Impact of AI assistance on incidental learning. In 27th international conference on intelligent user interfaces. 794– 806.Not checked yet

  Not checked in the free run.

• cites [45] Miller Tim. 2023. Explainable AI is Dead, Long Live Explainable AI! Hypothesis-driven Decision Support using Evaluative AI. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 333– 342.Not checked yet

  Not checked in the free run.

• cites [83] Zhang Zelun Tony, Feger Sebastian S, Dullenkopf Lucas, Liao Rulu, Süsslin Lukas, Liu Yuanting, and Butz Andreas. 2024. Beyond Recommendations: From Backward to Forward AI Support of Pilots’ Decision-Making Process. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1– 32.39286336Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing), S4 (based).

Other types of information synthesis may prompt critical thinking and reflection more effectively, making recommendations unnecessary.

“Other types of information synthesis may prompt critical thinking and reflection more effectively, making recommendations unnecessary.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The first study phase was approved by the Institutional Review Board at the primary research site.

“This study phase was approved by the Institutional Review Board at our primary research site.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants in all three studies were not domain experts and completed tasks that included student performance prediction, spatial reasoning, count estimation, and medication selection.

“Participants in all three studies were not domain experts and completed tasks that included student performance prediction [53], spatial reasoning [14], count estimation [14], and medication selection [63].”

• cites [53] Rastogi Charvi, Zhang Yunfeng, Wei Dennis, Varshney Kush R, Dhurandhar Amit, and Tomsett Richard. 2022. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1– 22.37360538Not checked yet

  Not checked in the free run.

• cites [14] Cao Shiye, Gomez Catalina, and Huang Chien-Ming. 2023. How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1– 26.Not checked yet

  Not checked in the free run.

• cites [63] Swaroop Siddharth, Buçinca Zana, Gajos Krzysztof Z, and Doshi-Velez Finale. 2024. Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure. In 29th International Conference on Intelligent User Interfaces (IUI’24). ACM.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing), S4 (based).

Because participants were experts trained in making decisions about LSIs, they may have been less likely to rely on the output of AI-enabled decision-support systems than non-experts.

“Second, because our participants were experts trained in making decisions about LSIs, they may have been less likely to rely on the output of AI-enabled decision-support systems than non-experts.”

Classified as probably not a substantial claim.

Because cognitive forcing strategies generally force decision makers to slow down, they may have trade-offs in time-constrained settings.

“However, because these strategies generally force decision makers to slow down, they may have trade-offs in time-constrained settings.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Rastogi et al. found participants were more likely to sufficiently shift away from incorrect AI-generated risk predictions when given more time to complete the task.

“Rastogi et al. [53] provided participants with AI-generated risk predictions at the start of a task, finding that participants were more likely to sufficiently shift away from incorrect predictions when given more time to complete the task.”

• cites [53] Rastogi Charvi, Zhang Yunfeng, Wei Dennis, Varshney Kush R, Dhurandhar Amit, and Tomsett Richard. 2022. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1– 22.37360538Not checked yet

  Not checked in the free run.

Classified as probably not a substantial claim.

Providing AI recommendations before an initial decision may be better for accuracy-time trade-offs when decisions are made under time pressure.

“This finding suggests that providing AI recommendations before an initial decision may be better for accuracy-time trade-offs when decisions are made under time pressure.”

• cites [63] Swaroop Siddharth, Buçinca Zana, Gajos Krzysztof Z, and Doshi-Velez Finale. 2024. Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure. In 29th International Conference on Intelligent User Interfaces (IUI’24). ACM.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

One participant's data were excluded from all 12 vignettes due to technical difficulties, but their debriefing responses were retained.

“As a result, we excluded data for this participant from all 12 vignettes but still considered their responses to the debriefing questions.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The findings and themes from the interview analysis were further discussed and refined through discussions with the entire research team, which includes HCI researchers and medical providers.

“The findings and themes from the analysis were further discussed and refined through discussions with the entire research team, which includes HCI researchers and medical providers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Process-oriented support proposes that recommendations may be provided at later points in events to confirm or challenge the initial decisions of users.

“Process-oriented support proposes that recommendations may be provided at later points in events to confirm or challenge the initial decisions of users [83].”

• cites [83] Zhang Zelun Tony, Feger Sebastian S, Dullenkopf Lucas, Liao Rulu, Süsslin Lukas, Liu Yuanting, and Butz Andreas. 2024. Beyond Recommendations: From Backward to Forward AI Support of Pilots’ Decision-Making Process. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1– 32.39286336Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Recommendations made later in the event with more information may be more accurate but less likely to affect decision making and the goals pursued by the team.

“Recommendations made later in the event with more information may be more accurate but less likely to affect decision making and the goals pursued by the team.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The second study phase was approved by the university's Institutional Review Board.

“This study phase was approved by our university’s Institutional Review Board.”

Classified as probably not a substantial claim. May combine two claims.

Need for blood transfusion was adjudicated by two physicians independently, and cases with disagreement were removed.

“Two physicians, one from surgery and one from emergency medicine, independently reviewed the chart of each patient to determine the need for blood transfusion. Four potential cases were removed because of a disagreement between the two physicians.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Because the earlier survey was anonymized, the authors do not know if interview participants had also completed the survey.

“Because the earlier survey was anonymized, we do not know if interview participants had also completed the survey.”

Probably not a substantial claim: may not pass S2 (checkable).

Process-oriented support may be especially well-suited for medical emergencies.

“Process-oriented support may be especially well-suited for medical emergencies.”

Classified as probably not a substantial claim.

Participants were recruited with Advanced Trauma Life Support certification to ensure they were trained in making decisions about LSIs during trauma resuscitation.

“We recruited physicians and advanced practice providers who had completed Advanced Trauma Life Support certification to ensure that participants were trained in making decisions about LSIs during trauma resuscitation.”

May combine two claims.

The simulated patient vignettes were designed with clinician collaboration and four pilots to ensure ecological validity.

“To design the simulated patient vignettes, we collaborated with clinicians on our team, ensuring the ecological validity of the experiment. We also conducted four pilots (two with medical students with experience in trauma resuscitation and two with advanced practice providers), iteratively resolving issues.”

The quote only partly states this claim.

Participants might have been hesitant to accept recommendations from the system due to concerns about over-reliance.

“As a result, participants might have been hesitant to accept recommendations from the system.”

Probably not a substantial claim: may not pass S3 (load-bearing).

When under time pressure, AI-before support led to faster response times than other support types with similar levels of accuracy.

“When under time pressure, AI-before support led to faster response times than the other support types with similar levels of accuracy.”

• cites [63] Swaroop Siddharth, Buçinca Zana, Gajos Krzysztof Z, and Doshi-Velez Finale. 2024. Accuracy-Time Tradeoffs in AI-Assisted Decision Making under Time Pressure. In 29th International Conference on Intelligent User Interfaces (IUI’24). ACM.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Both studies suggest that time pressure may have contributed to anchoring, with participants less likely to deviate from recommendations provided at the start of a task or from initial decisions when recommendations were provided later.

“Both studies suggest that time pressure may have contributed to anchoring, with participants less likely to deviate from recommendations provided at the start of a task [53] or from initial decisions when recommendations were provided later [14].”

• cites [53] Rastogi Charvi, Zhang Yunfeng, Wei Dennis, Varshney Kush R, Dhurandhar Amit, and Tomsett Richard. 2022. Deciding fast and slow: The role of cognitive biases in ai-assisted decision-making. Proceedings of the ACM on Human-Computer Interaction 6, CSCW1 (2022), 1– 22.37360538Not checked yet

  Not checked in the free run.

• cites [14] Cao Shiye, Gomez Catalina, and Huang Chien-Ming. 2023. How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1– 26.Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable).

If delays in provider decision making are leading to delays in intervention performance and patient deterioration, recommendations may need to be provided before initial decisions are made by providers.

“Finally, if delays in provider decision making are leading to delays in intervention performance and patient deterioration, recommendations may need to be provided before initial decisions are made by providers.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Cao et al. found participants were more likely to shift from their initial decision and accept AI recommendations when given more decision-making time.

“In contrast, Cao et al. [14] provided participants with AI recommendations after they had made initial decisions. Participants were more likely to shift from their initial decision and accept AI recommendations when given more decision-making time.”

• cites [14] Cao Shiye, Gomez Catalina, and Huang Chien-Ming. 2023. How Time Pressure in Different Phases of Decision-Making Influences Human-AI Collaboration. Proceedings of the ACM on Human-Computer Interaction 7, CSCW2 (2023), 1– 26.Not checked yet

  Not checked in the free run.

May combine two claims.

Recommendations made earlier in the medical event may be more likely to affect decision making and prevent teams from pursuing incorrect goals, but may have lower accuracy because they are generated with limited information.

“Recommendations made by the system earlier in the medical event may be more likely to affect decision making and prevent teams from pursuing incorrect goals (e.g., preparing blood transfusion for a patient that does not need one). However, these recommendations may have lower accuracy because they are generated with limited information.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Decision support could function as a coordination mechanism, aligning physicians from different specialties and experience levels on a treatment plan.

“Prior research has suggested that decision support could function as a coordination mechanism, aligning physicians from different specialties and experience levels on a treatment plan [44].”

• cites [44] Mastrianni Angela, Almengor Lynn, and Sarcevic Aleksandra. 2022. Alerts as coordination mechanisms: implications for designing alerts for multidisciplinary and shared decision making. Proceedings of the ACM on Human-Computer Interaction 6, GROUP (2022), 1– 14.37360538Not checked yet

  Not checked in the free run.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Other types of information-driven AI support may also assist human decision makers in managing accuracy-time trade-offs, especially during dynamic events.

“However, other types of information-driven AI support may also assist human decision makers in managing accuracy-time trade-offs, especially during dynamic events.”

May combine two claims.

Although only 35 participants completed the experiment, a total of 816 decision-making instances were collected.

“Although only 35 participants completed the experiment, we collected a total of 816 decision-making instances.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Agency, autonomy, and authority could be further reduced if AI recommendations are included in archival systems that can be used outside the medical event (e.g., as evidence in a medical malpractice lawsuit).

“Agency, autonomy, and authority could be further reduced if AI recommendations are included in archival systems that can be used outside the medical event (e.g., as evidence in a medical malpractice lawsuit).”

• AssumptionRests on an assumption: Providers are legally liable for their decisions.

Probably not a substantial claim: may not pass S3 (load-bearing).

To evaluate the effectiveness of different human-AI interaction strategies, an online experiment was used instead of in-person simulations.

“Third, to evaluate the effectiveness of different human-AI interaction strategies, we used an online experiment instead of in-person simulations.”

May combine two claims.

The design research phase included only participants from the primary research site and most were emergency medicine physicians.

“First, the design research phase included only participants from our primary research site and most were emergency medicine physicians.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Although developed from research on decision support for trauma resuscitation, the implications may also apply to other time-critical events, such as firefighting and aviation.

“Although we developed these implications from our research on decision support for trauma resuscitation, they may also apply to other time-critical events, such as firefighting and aviation [80, 83].”

• cites [80] Zhang Kexin, Cochran Brianna R, Chen Ruijia, Hartung Lance, Sprecher Bryce, Tredinnick Ross, Ponto Kevin, Banerjee Suman, and Zhao Yuhang. 2024. Exploring the Design Space of Optical See-through AR Head-Mounted Displays to Support First Responders in the Field. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 1– 19.Not checked yet

  Not checked in the free run.

• cites [83] Zhang Zelun Tony, Feger Sebastian S, Dullenkopf Lucas, Liao Rulu, Süsslin Lukas, Liu Yuanting, and Butz Andreas. 2024. Beyond Recommendations: From Backward to Forward AI Support of Pilots’ Decision-Making Process. Proceedings of the ACM on Human-Computer Interaction 8, CSCW2 (2024), 1– 32.39286336Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• AssumptionRests on an assumption: In time-critical contexts such as firefighting and aviation, teams frequently make decisions with limited information under time pressure.

Probably not a substantial claim: may not pass S3 (load-bearing).

Because different hospitals may have different approaches to treating patients with life-threatening injuries, conducting the online experiment with participants from five other hospital systems outside the primary research site allowed collecting a wider set of perspectives.

“Because different hospitals may have different approaches to treating patients with life-threatening injuries, conducting the online experiment with participants from five other hospital systems outside of our primary research site allowed us to collect a wider set of perspectives.”

• AssumptionRests on an assumption: Different hospitals may have different approaches to treating patients with life-threatening injuries.

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing).

Online experiments are valuable because they can provide initial insights into the effectiveness of different human-AI interaction strategies before conducting in-person simulations, which are more resource- and time-intensive.

“Online experiments are valuable because they can provide initial insights into the effectiveness of different human-AI interaction strategies before conducting in-person simulations, which are more resource- and time-intensive.”

May combine two claims.

Although providing AI information and recommendations improved decisions about life-saving interventions, two socio-technical barriers to implementing AI recommendations for time-critical decision making were identified.

“Although we found that providing AI information and recommendations improved decisions about life-saving interventions, we also identified two socio-technical barriers to implementing AI recommendations for time-critical decision making.”