An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study

Classified as probably not a substantial claim.

Poor control, defined as a hemoglobin A1c level >9.0%, contributes to complications, including nephropathy, retinopathy, and neuropathy.

“Poor control, defined as a hemoglobin A1c (HbA1c) level >9.0%, contributes to complications, including nephropathy [1-6], retinopathy [4,7], and neuropathy [4,8].”

• cites [1] sourcePartly supported

  The Results (p. 2) show that intensive glucose control reduced nephropathy, supporting a link between glycemic control and that complication. The paper does not define poor control as HbA1c >9.0%; it reports no significant effect on retinopathy (Results, p. 2), and its secondary-outcomes table (p. 10) does not show a reduction in neuropathy. (LLM, conf=0.90)

  Experimental — accuracy not yet measured
• cites [2] sourceNot checked — no full text of the source
• cites [3] sourcePartly supported

  The Discussion says glucose control may reduce microvascular complications, and the Microvascular Results report less worsening of albumin excretion with intensive treatment. But the paper does not define poor control as HbA1c >9.0% or establish that this threshold contributes to nephropathy, retinopathy, and neuropathy; it reports no significant treatment-group differences in those three outcomes. Page numbers were not available in our copy. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [4] sourcePartly supported

  The Discussion’s summary of main results reports that intensive glucose control reduces the risk of developing retinopathy, nephropathy, and neuropathy compared with conventional treatment. But the review does not define poor control as HbA1c >9.0% or establish that threshold for these complications; its Figure 4 uses 9.0% to stratify risk of severe hypoglycaemia. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [5] sourceNot checked — no full text of the source
• cites [6] sourcePartly supported

  The Introduction reports an association between poor glycemic control and microvascular complications in type 2 diabetes, and the Results > Outcomes section finds that intensive control reduced albuminuria. The paper does not define poor control as HbA1c >9.0% or establish that poor control causes nephropathy; it also finds no demonstrated reduction in clinical renal endpoints. This citation is attached to nephropathy, not to the sentence’s separate retinopathy and neuropathy claims. (LLM, conf=0.88)

  Experimental — accuracy not yet measured
• cites [7] sourceNot checked — no full text of the source
• cites [8] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S3 (load-bearing).

In federally qualified health centers, the percentage with poor control was 32% in 2016, up from 29% in 2009.

“In federally qualified health centers (FQHCs), the percentage with poor control was 32% in 2016 (up from 29% in 2009)”

• cites [13] sourcePartly supported

  The Abstract identifies 2009 data, and the Results report that 71.0% of diabetic patients had controlled HbA1c levels—implying 29.0% did not meet that control measure. The paper provides no 2016 figure, so it does not support the claimed increase to 32%. (LLM, conf=0.87)

  Experimental — accuracy not yet measured
• cites [14] National Health Center Program Uniform Data System (UDS) Awardee DataHealth Resources & Services Administration2018-02-22https://bphc.hrsa.gov/uds/datacenter.aspx#fn8Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Most of these studies used quantitative methods and examined tools that have already been developed.

“Most of these studies used quantitative methods and examined tools that have already been developed [35-37].”

• cites [35] sourcePartly supported

  The Results, Summary of Included Studies (page unavailable), says that most of the 48 studies evaluated AI from the user perspective, which supports the claim that they examined tools. It does not establish that most used quantitative methods or that the tools had already been developed; it also describes studies focused on algorithm development. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [36] sourcePartly supported

  The Results (Study Characteristics) report that 7 of 31 studies used qualitative methods, supporting the claim that most used other, largely quantitative designs. The Results (Risk-of-Bias and Quality Assessments) also describe some studies developing and initially evaluating agents; the review does not establish that most examined tools developed before the studies began. Page numbers are unavailable in our copy. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [37] sourcePartly supported

  The Methods (Data Analysis; page unavailable) report quantitative usability statistics for an app that participants used, supporting the claim that this study examined a functioning tool. But the same section reports qualitative interview analysis, and Methods (Development of GUI; page unavailable) describes developing and iteratively refining the app. This paper therefore does not support characterizing its approach as solely quantitative or its tool as already developed independently of the study. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim.

Researchers and clinicians are using artificial intelligence to integrate electronic health records and social risk factors, such as neighborhood characteristics, to predict outcomes important to individuals with diabetes, including poor control.

“researchers and clinicians are using artificial intelligence (AI) to integrate electronic health records (EHRs) and social risk factors, such as neighborhood characteristics, to predict outcomes important to individuals with diabetes, including poor control [17-22].”

• cites [17] sourceNot checked — no full text of the source
• cites [18] sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [19] sourcePartly supported

  Split into 2 atoms — 0 confirmed, 2 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured
• cites [20] DevireddyLDunnDShermanMA feature-selection based approach for the detection of diabetes in electronic health record dataMichael W Sherman2014572023-05-17http://www.michaelwsherman.com/projects/diabetes/report.pdfNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [21] Berkman, LF, 2014, Social Epidemiology (2 edition)Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [22] sourcePartly supported

  Split into 2 atoms — 0 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 1 unverifiable. Expand the claim to see per-atom verdicts.

  Experimental — accuracy not yet measured

Classified as probably not a substantial claim. May combine two claims.

Sociotechnical theory posits that the implementation of technology depends on values, mindsets, and communication and is an evolutionary process best achieved by early and active engagement with frontline workers.

“The sociotechnical theory posits that the implementation of technology depends on values, mindsets, and communication and is an evolutionary process best achieved by early and active engagement with frontline workers [33,34].”

• cites [33] sourceNot checked — no full text of the source
• cites [34] sourceSupported

May combine two claims.

The implementation of AI tools for diabetes has lagged, and few tools are used in practice, limiting their impact.

“Unfortunately, the implementation of AI tools for diabetes has lagged, and few tools are used in practice, limiting their impact.”

• cites [31] sourcePartly supported

  The Introduction (Background; page not provided) says health care AI implementation is not widespread and that its potential depends on integration into routine workflows. The Results (AI Application Characteristics; page not provided) identify four studies of diabetic-retinopathy applications, but the review does not establish that few diabetes tools are used in practice or measure whether limited use has reduced their impact. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

These applications used computer vision to diagnose diabetic retinopathy from retinal images and electronic health record data to predict those at risk for hyperglycemia.

“These applications used computer vision to diagnose diabetic retinopathy from retinal images and EHR data to predict those at risk for hyperglycemia.”

• cites [31] sourcePartly supported

  The Results > Evaluation Outcomes section (page unavailable) reports an AI diagnostic system for diabetic retinopathy and gives its imageability rate, supporting the retinopathy-detection part of the claim. It does not specify computer vision or retinal images in that passage. Searches for hyperglycemia and EHR-based risk prediction found no passage supporting the second part. (LLM, conf=0.82)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

The Technology Acceptance Model explains approximately 40% of the variance in individuals' intention to use a new technology and actual use.

“The model explains approximately 40% of the variance in individuals’ intention to use a new technology and actual use [39].”

• cites [39] sourceSupported

Classified as probably not a substantial claim.

The Technology Acceptance Model theorizes that individuals' intention to use new technology is determined by perceived usefulness and perceived ease of use.

“It theorizes that individuals’ intention to use new technology is determined by perceived usefulness, defined as “the extent to which a person believes that using [a new technology] will enhance his or her job performance,” and perceived ease of use, defined as “the degree to which a person believes that using [a new technology] will be free of effort” [39].”

• cites [39] sourceSupported

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Usability of and satisfaction with AI tools are generally high.

“researchers are starting to pay attention to end users [35] and are finding that usability of and satisfaction with AI tools are generally high [35-37].”

• cites [35] sourcePartly supported

  The Results’ Summary of Included Studies reports that most patients found one AI symptom checker easy to use and that 19 of 20 children found a chatbot easy to use. These examples support positive usability findings, but the review does not establish that usability and satisfaction with AI tools are generally high across the studies it covers; its Perception, Usability, Workload, and Trust section also describes usability barriers. No page numbers were provided in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [36] sourcePartly supported

  The Results (Overall Evaluation of Conversational Agents) report positive or mixed evidence for perceived ease of use or usefulness in 27/30 studies and satisfaction in 26/31. This supports a generally favorable assessment, but the review covers health care conversational agents—not AI tools broadly—and combines positive with mixed results rather than establishing that ratings were generally high. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [37] sourcePartly supported

  The Results (Satisfaction) report a mean usability score of 91.94, rated “excellent,” and the Discussion (Principal Findings) says clinicians found the app intuitive and easy to use. The study assessed one COVID-19 prediction app at a single hospital, so it does not establish that usability and satisfaction are generally high across AI tools. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The number of Americans with poorly controlled diabetes has been increasing, contributing to preventable morbidity and mortality.

“The number of Americans with poorly controlled diabetes has been increasing, contributing to preventable morbidity and mortality [10-12].”

• cites [10] Comprehensive Diabetes Care (CDC)NCQA2017-11-29http://www.ncqa.org/report-cards/health-plans/state-of-health-care-quality/2016-table-of-contents/diabetes-careNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [11] Quality: Traditional MIPS RequirementsQuality Payment Program2017-11-29https://qpp.cms.gov/mips/quality-measuresNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [12] sourcePartly supported

  The Introduction (page not available in our copy) says that type 2 diabetes prevalence is growing and that poor glycemic control increases the risk of complications, which supports the morbidity concern. It does not establish that the number of Americans with poorly controlled diabetes is increasing or specifically link poor control to mortality. (LLM, conf=0.85)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing).

Communities with poor housing, transportation, poverty, and education have higher rates of diabetes.

“communities with poor housing, transportation, poverty, and education have higher rates of diabetes [23-25].”

• cites [23] sourceNot checked — no full text of the source
• cites [24] sourceSupported
• cites [25] sourcePartly supported

  The Introduction and Discussion (pages unavailable) report an association between neighborhood deprivation and type 2 diabetes. The Methods define deprivation using low income and low education, among other indicators, but do not include poor housing or transportation. The source therefore supports the poverty and education aspects, not the full list of community conditions. (LLM, conf=0.87)

  Experimental — accuracy not yet measured

May combine two claims.

Our respondents asked for steps that could be taken in response to predictions and wanted to know that those actions would lead to better health, echoing the sentiment found in other studies.

“Although usability and accuracy were deemed important, our respondents asked for steps that could be taken in response to predictions and wanted to know that those actions would lead to better health, echoing the sentiment found in other studies [35].”

• cites [35] sourcePartly supported

  In Results > Summary of Included Studies (page unavailable), the review reports that tailored recommendations were important to stakeholders designing a risk-prediction tool and that some patients lacked information about self-management. This supports the desire for steps to take, but our copy does not report that users wanted assurance those steps would lead to better health. (LLM, conf=0.79)

  Experimental — accuracy not yet measured

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

For many years, researchers and policy makers have issued warnings regarding the black-box nature of AI and its role in widening disparities.

“For many years, researchers and policy makers have issued warnings regarding the black-box nature of AI and its role in widening disparities [43,44].”

• cites [43] sourcePartly supported

  The introduction (p. 2) describes growing concern that algorithms may reproduce disparities and says proprietary algorithms are difficult for independent researchers to examine. It does not establish that both researchers and policymakers have issued warnings for many years. (LLM, conf=0.84)

  Experimental — accuracy not yet measured
• cites [44] FjeldJAchtenNHilligossHNagyASrikumarMPrincipled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AIBerkman Klein Center for Internet & Society20202023-05-17https://dash.harvard.edu/handle/1/42160420Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

Other clinicians have responded favorably to the usability of tools that use AI.

“For example, similar to our results, other clinicians have responded favorably to the usability of tools that use AI [36,37].”

• cites [36] sourceSupported
• cites [37] sourceSupported

May combine two claims.

21/22 (95%) participants said responding to the tool in a comprehensive manner requires the engagement of a comprehensive team; there was strong consensus regarding the role of clinicians and nurses, but ambivalence regarding administrators and patients.

“Responding to the tool in a comprehensive manner requires the engagement of a comprehensive team. Although there was strong consensus regarding the role of clinicians and nurses, participants expressed ambivalence regarding administrators and patients.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

The quote only partly states this claim.

5/22 (23%) participants were concerned that the tool's accuracy and implementation were not supported by evidence.

“Participants were concerned that the tool’s accuracy and implementation were not supported by evidence.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 22 participants, 14 (63%) were physicians.

“Two-thirds (14/22, 63%) of respondents were physicians.”

Probably not a substantial claim: may not pass S3 (load-bearing).

2/22 (9%) participants said the ability to customize the tool is important because implementation could differ across practices and clinicians.

“The ability to customize the tool is important because implementation could differ across practices and clinicians.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

8/22 (36%) participants said successful implementation would require trialability, training, interoperability, and buy-in.

“Successful implementation would require trialability, training, interoperability, and buy-in.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Other participants argued that the tool would facilitate the delivery of proactive care, building on the core function of primary care.

“Other participants argued that the tool would facilitate the delivery of proactive care, building on the core function of primary care”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Implementation would require buy-in and need to be tailored to the demands and resources of clinics and communities.

“implementation would require buy-in and need to be tailored to the demands and resources of clinics and communities”

• Also stated in: § Overview · hedged

  “Implementation would require buy-in and need to be tailored to the demands and resources of clinics and communities”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

In total, 22 individuals participated in the study.

“In total, 22 individuals participated in the study.”

• Also stated in: § Overview · plain

  “In total, 22 individuals participated in this study.”

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

19/22 (86%) participants recommended integrating functionality and relevant information from within the EHR.

“Participants recommended integrating functionality and relevant information from within the EHRa.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims.

20/22 (91%) participants wanted to share this information with patients to empower them and support transparency but were also concerned that the information would cause confusion and stress.

“Participants wanted to share this information with patients to empower them and support transparency but were also concerned that the information would cause confusion and stress.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims.

Adoption of the tool is dependent on its validation, transparency, actionability, and design and could be increased with changes to the interface and usability.

“adoption of the tool is dependent on its validation, transparency, actionability, and design and could be increased with changes to the interface and usability”

• Also stated in: § Overview · hedged

  “Adoption of the tool is dependent on its validation, transparency, actionability, and design and could be increased with changes to the interface and usability”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Classified as probably not a substantial claim. May combine two claims. The quote only partly states this claim.

6/22 (27%) participants said to act on the information, clinicians and staff need to understand which risk factors are modifiable and which actions will have the greatest impact on lowering risk.

“To act on the information, clinicians and staff need to understand which risk factors are modifiable and which actions will have the greatest impact on lowering risk.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

7/22 (32%) participants noted that such a tool does not exist and that it would fill a gap.

“several participants (7/22, 32%) noted that such a tool does not exist and that it would fill a gap”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

22/22 (100%) participants recommended other events that could be predicted, including cardiovascular disease, uncontrolled hypertension, worsening depression, care gaps, and missed appointments.

“Participants recommended other events that could be predicted, including cardiovascular disease, uncontrolled hypertension, worsening depression, care gaps (eg, preventive services), and missed appointments.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims.

17/22 (77%) participants lacked consensus regarding when the alert should appear, with some wanting it at the point of care and others wanting to review the information outside of visits.

“There was a lack of consensus regarding when the alert should appear, with some wanting it at the point of care, whereas others wanted to review the information outside of visits (eg, periodic lists or a dashboard).”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims.

12/22 (55%) participants said the local context affects what can be done in response to the information provided by the tool, and participants will become frustrated if the tool recommends an option that is not available.

“The local context affects what can be done in response to the information provided by the tool. Conversely, participants will become frustrated if the tool recommends an option that is not available.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims. The quote only partly states this claim.

8/22 (36%) participants thought the utility is limited for clinicians who know their patients well or have access to existing programs, and some would rather focus on people who are already uncontrolled.

“The utility is limited for those clinicians who know their patients well or have access to existing programs, and some would rather focus on people who are already uncontrolled.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims.

The mean number of people (out of 1000) whose prognosis the tool would need to accurately predict for participants to consider using it was 617 (SD 264), with responses ranging from 20 to 900.

“the mean number of people whose prognosis the tool would need to accurately predict was 617 (SD 264), although the responses ranged from 20 to 900.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

4/22 (18%) participants said the tool needs to be validated against patient-oriented outcomes so that clinics can quantify the potential return on their investment.

“The tool needs to be validated against patient-oriented outcomes so that clinics can quantify the potential return on their investment.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

May combine two claims. The quote only partly states this claim.

15/22 (68%) participants were concerned the tool would lead to harms, contribute to overdiagnosis, be used punitively, and make care more expensive.

“Participants were concerned the tool would lead to harms, contribute to overdiagnosis, be used punitively, and make care more expensive.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Classified as probably not a substantial claim. May combine two claims.

The tool has the potential to be useful because it provides information that is not currently available and can make care more efficient and effective.

“this tool has the potential to be useful because it provides information that is not currently available and can make care more efficient and effective”

• Also stated in: § Theme 1 · plain

  “The tool has the potential to be useful because it provides information that is not currently available and can make care more efficient and effective.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 22 participants, 6 (27%) worked in federally qualified health centers.

“whereas 27% (6/22) of respondents worked in federally qualified health centers.”

The quote only partly states this claim.

14/22 (64%) participants were concerned that the tool would exacerbate existing problems, such as health disparities and alert fatigue.

“Participants were concerned that the tool would exacerbate existing problems, such as health disparities and alert fatigue.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing). The quote only partly states this claim.

13/22 (59%) participants said using user-centered design principles has the potential to minimize the tool's impact on workflows and maximize readability.

“Using user-centered design principles has the potential to minimize the tool’s impact on workflows and maximize readability.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

Of the 22 participants, 8 (36%) worked in academic health centers.

“Overall, 36% (8/22) of respondents worked in academic health centers”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants believed that perceived usability and readability would be key drivers of adoption (13/22, 59%).

“Participants believed that perceived usability and readability would be key drivers of adoption (13/22, 59%):”

May combine two claims.

Participants wanted to understand which risk factors are modifiable and which actions will have the greatest impact on lowering risk, in addition to knowing why someone is at high risk (6/22, 27%).

“Knowing why someone is at high risk is necessary but insufficient. Participants also wanted to understand which risk factors are modifiable and which actions will have the greatest impact on lowering risk (6/22, 27%):”

May combine two claims.

Clinicians who know their patients well or have access to existing programs thought the tool's utility was limited, and some would rather focus on people who are already uncontrolled (8/22, 36%).

“In particular, those clinicians who know their patients well or have access to existing programs thought the utility was limited, and some would rather focus on people who are already uncontrolled (8/22, 36%):”

Classified as probably not a substantial claim. May combine two claims.

Clinicians and staff believed that using the tool would increase their focus on diabetes by scheduling more frequent visits, interacting with patients between visits, managing diabetes even when acute issues emerge, and providing targeted education (20/22, 91%).

“As a result of using the tool, clinicians and staff thought they would increase their focus on diabetes by scheduling more frequent visits, interacting with patients in between visits, managing diabetes even when acute issues emerge, and providing targeted education (20/22, 91%):”

May combine two claims.

Participants were ambivalent about the tool's impact on susceptible populations: some thought these patients needed attention most, while others thought making a positive impact would be difficult (7/22, 32%).

“Participants were ambivalent about the tool’s impact on susceptible populations. Some participants thought that these were the patients who needed attention the most, whereas others thought that making a positive impact would be difficult (7/22, 32%):”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The sample was predominantly women, Hispanic, and physicians.

“They were predominantly women, Hispanic, and physicians (Table 1).”

May combine two claims.

Participants expected a degree of transparency and wanted to know how the tool was developed and the rationale behind the high risk of an individual, which allows them to gauge the tool's credibility (11/22, 50%).

“The participants expected a degree of transparency and wanted to know how the tool was developed and the rationale behind the high risk of an individual. This information allows them to gauge the tool’s credibility (11/22, 50%):”

Probably not a substantial claim: may not pass S1 (asserted), S3 (load-bearing).

Participants believed the tool needs to be validated against patient-oriented outcomes so that clinics can quantify the potential return on investment (4/22, 18%).

“The tool needs to be validated against patient-oriented outcomes so that clinics can quantify the potential return on investment (4/22, 18%):”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants believed that by integrating information about mental health and social risk factors, the tool would facilitate individualized, holistic care (11/22, 50%).

“By integrating information about mental health and social risk factors, our participants (11/22, 50%) believed that the tool would facilitate individualized, holistic care:”

Probably not a substantial claim: may not pass S3 (load-bearing).

The sample included a nurse practitioner, physician assistant, behavioral therapist, and social worker.

“The sample also included a nurse practitioner, physician assistant, behavioral therapist, and social worker.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Others believed the tool could be used to improve elements of population health, such as improving the quality of care delivered and allocating resources to high-need patients (10/22, 45%).

“Others believed that the tool could be used to improve elements of population health, such as improving the quality of care delivered and allocating resources to high-need patients (10/22, 45%):”

May combine two claims.

Participants were concerned that the tool would lead to harms, contribute to overdiagnosis, be used punitively, and make care more expensive (15/22, 68%).

“Participants had myriad concerns about the tool. First, they were concerned that the tool would lead to harms, contribute to overdiagnosis, be used punitively, and make care more expensive (15/22, 68%):”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants were concerned that the tool would exacerbate existing issues such as health disparities and alert fatigue.

“They were also concerned that the tool would exacerbate existing issues such as health disparities and alert fatigue:”

Classified as probably not a substantial claim.

Respondents wanted to be able to tailor the tool to their local environment, noting that the suggestions offered and the recipients of the information needed to be customized to the resources, needs, and workflows of their unique clinics.

“Finally, they wanted to be able to tailor the tool to their local environment, noting that the suggestions offered and the recipients of the information needed to be customized to the resources, needs, and workflows of their unique clinics.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Successful implementation would require trialability, training, interoperability, and buy-in (8/22, 36%).

“Successful implementation would require trialability, training, interoperability, and buy-in (8/22, 36%):”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Participants thought the model could be applied to other conditions and recommended the tool be used to predict important events in primary care, including cardiovascular disease, uncontrolled hypertension, worsening depression, care gaps, and missed appointments (22/22, 100%).

“Participants thought that this model could be applied to other conditions and recommended that the tool be used to predict important events in primary care, including cardiovascular disease, uncontrolled hypertension, worsening depression, care gaps (eg, preventive services), and missed appointments (22/22, 100%):”

Classified as probably not a substantial claim. May combine two claims.

Participants believed that the tool could improve population health and contribute to individualized care.

“They also believed that the tool could improve population health and contribute to individualized care.”

May combine two claims.

Participants wanted to share information with patients to empower them and support transparency but were concerned that the information would cause confusion and stress (20/22, 91%).

“Participants wanted to share this information with patients to empower them and support transparency but were also concerned that the information would cause confusion and stress (20/22, 91%).”

May combine two claims.

They are concerned about how AI can perpetuate the racial biases embedded within data sets and about their role in supporting biased systems.

“They are concerned about how AI can perpetuate the racial biases embedded within data sets and about their role in supporting biased systems.”

May combine two claims.

During the interviews, participants noted that the tool is not available now and would generally change their behavior.

“During the interviews, they noted that the tool is not available now and would generally change their behavior.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Others were concerned that AI would perpetuate biases and that alert fatigue would contribute to burnout.

“Others were concerned that AI would perpetuate biases and that alert fatigue would contribute to burnout.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants believed the ability to customize the tool is important because implementation could differ across practices and clinicians (2/22, 9%).

“The ability to customize the tool is important because implementation could differ across practices and clinicians (2/22, 9%):”

Probably not a substantial claim: may not pass S3 (load-bearing).

Participants recommended integrating functionality and relevant information within the EHR (19/22, 86%).

“Participants recommended integrating functionality and relevant information within the EHR (19/22, 86%).”

Classified as probably not a substantial claim.

To gauge the tool's credibility, participants wanted to know why the patients were at high risk and what they could do to reduce that risk.

“To gauge the tool’s credibility, they wanted to know why the patients were at high risk and what they could do to reduce that risk.”

May combine two claims.

Participants were concerned about alert fatigue, bias, and transparency.

“Conversely, participants were concerned about alert fatigue, bias, and transparency.”

May combine two claims.

The tool was not uniformly accepted; several respondents indicated that it did not provide useful information for patients well known to the practice and for practices already offering comprehensive services.

“Despite these benefits, the tool was not uniformly accepted, with several respondents indicating that it did not provide useful information for those patients who are well known to the practice and for those practices already offering comprehensive services.”

Classified as probably not a substantial claim. May combine two claims.

Participants thought that information without context could be harmful and that they would need scripts to explain the results in a patient-centric manner.

“They thought that the information without context could be harmful and that they would need scripts to explain the results in a patient-centric manner:”

May combine two claims.

From the surveys, respondents found the tool to be useful and easy to use and, if available, would use it.

“From the surveys, respondents found the tool to be useful and easy to use and, if available, would use it.”

May combine two claims.

Most participants found the tool to be easy to use and useful.

“Most participants found the tool to be easy to use and useful.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

All members of the primary care team have potential roles to play, including front desk personnel, pharmacists, and social workers, and the recipients of the information may be practice dependent.

“All members of the primary care team have potential roles to play, including front desk personnel, pharmacists, and social workers. As roles differ for each practice, the recipients of the information may be practice dependent:”

May combine two claims.

The local context affects what can be performed in response to the information provided by the tool, and participants will become frustrated if the tool recommends an option that is not available (12/22, 55%).

“The local context affects what can be performed in response to the information provided by the tool. Conversely, participants will become frustrated if the tool recommends an option that is not available (12/22, 55%):”

May combine two claims.

To enhance adoption, respondents wanted to know why the patients were at risk and what could be done to reduce that risk.

“To enhance adoption, respondents wanted to know why the patients were at risk and what could be done to reduce that risk.”

May combine two claims.

The clinicians and staff members in our study called for greater explainability, wanted these issues explicitly acknowledged and addressed, and cautioned that these tools will continue to languish on shelves in the absence of satisfactory solutions.

“The clinicians and staff members in our study called for greater explainability (ie, justifications for the tool’s output), wanted these issues explicitly acknowledged and addressed, and cautioned that these tools will continue to languish on shelves in the absence of satisfactory solutions [44].”

• cites [44] FjeldJAchtenNHilligossHNagyASrikumarMPrincipled artificial intelligence: mapping consensus in ethical and rights-based approaches to principles for AIBerkman Klein Center for Internet & Society20202023-05-17https://dash.harvard.edu/handle/1/42160420Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

May combine two claims.

Responding to the tool in a comprehensive manner requires the engagement of a comprehensive team; there was strong consensus regarding the role of clinicians and nurses, but ambivalence regarding administrators and patients (21/22, 95%).

“Responding to the tool in a comprehensive manner requires the engagement of a comprehensive team. Although there was strong consensus regarding the role of clinicians and nurses, participants expressed ambivalence regarding administrators and patients (21/22, 95%).”

May combine two claims.

Most participants found the tool to be easy to use and useful, although they had concerns about alert fatigue, bias, and transparency.

“Most participants found the tool to be easy to use and useful, although they had concerns about alert fatigue, bias, and transparency.”

• AssumptionRests on an assumption: The sample electronic health record alert and description of the AI tool were realistic enough that participants’ responses reflect how they would use an actual tool.
• AssumptionRests on an assumption: Participants’ responses about a hypothetical tool that has yet to be created can identify factors that affect adoption of the eventual tool.

Probably not a substantial claim: may not pass S3 (load-bearing).

The protocol was approved by the Institutional Review Board of the University of Houston (STUDY00002980).

“The protocol was approved by the Institutional Review Board of the University of Houston (STUDY00002980).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

The coders independently applied codes in a second reading and met to reconcile coding discrepancies and modify the codebook.

“Using the updated codebook, the coders independently applied codes to the interviews in a second reading and met to reconcile coding discrepancies and modify the codebook (version 2).”

May combine two claims.

In that study, only 14% (4/28) of users indicated that they would recommend the tool to others, and many users reported that the interventions were inappropriate or not useful.

“only 14% (4/28) of users indicated that they would recommend the tool to others, and many users reported that the interventions were inappropriate or not useful [32].”

• cites [32] sourceNot checked — no full text of the source

Probably not a substantial claim: may not pass S2 (checkable).

Taken together, these studies indicate that a greater focus on AI implementation and end-user engagement during development are needed to tailor tools to clinical resources and workflows.

“Taken together, these studies indicate that a greater focus on AI implementation and end-user engagement during development are needed to tailor tools to clinical resources and workflows.”

Probably not a substantial claim: may not pass S3 (load-bearing).

Digitization risks overwhelming clinicians.

“Although this digitization offers tremendous opportunities for prediction, it also risks overwhelming clinicians [27].”

• cites [27] Medscape lifestyle report 2016: bias and burnoutMedscape2021-03-22https://www.medscape.com/slideshow/lifestyle-2016-overview-6007335Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The national target for poor control in Healthy People 2030 is 11.6%.

“the metric has been included in Healthy People 2030, which sets the national target at 11.6%”

• cites [15] Reduce the proportion of adults with diabetes who have an A1c value above 9 percentOffice of Disease Prevention and Health Promotion2022-09-06https://tinyurl.com/dkx26uwxNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

• cites [16] Diabetes: Hemoglobin A1c (HbA1c) Poor Control (> 9%)Centers for Medicare & Medicaid Services2021-08-13https://qpp.cms.gov/docs/ecqm-specs/2017/EC_CMS122v5_NQF0059_Diab_HbA1c_Ctrl/CMS122v5.htmlNot matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Classified as probably not a substantial claim. May combine two claims.

Adoption of the tool is dependent on its validation, transparency, actionability, and design, and could be increased by changing the interface and usability.

“Adoption of the tool is dependent on its validation, transparency, actionability, and design and could be increased by changing the interface and usability.”

Probably not a substantial claim: may not pass S3 (load-bearing).

The amount of data available to clinicians has increased exponentially with the growth of electronic health records, remote patient monitoring, and geo-tracking.

“With the growth of EHRs, remote patient monitoring, and geo-tracking, the amount of data available to clinicians has increased exponentially [26].”

• cites [26] CulbertsonNThe skyrocketing volume of healthcare data makes privacy imperativeForbes2021862022-08-29https://www.forbes.com/sites/forbestechcouncil/2021/08/06/the-skyrocketing-volume-of-healthcare-data-makes-privacy-imperative/Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Probably not a substantial claim: may not pass S3 (load-bearing).

The transcripts were coded by 2 individuals using thematic content analysis in NVivo.

“The transcripts were coded by 2 individuals using thematic content analysis in NVivo (QSR International).”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Coding stopped once study objectives were saturated, indicating that no new information was identified.

“Coding stopped once study objectives were saturated, indicating that no new information was identified.”

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

We did not ask participants to use a prototype of the tool when responding to the questions; if they had, their responses regarding ease of use and usefulness may have been different.

“Second, we did not ask the participants to use a prototype of the tool when responding to the questions. If they had, their responses to the questions regarding ease of use and usefulness may have been different.”

Probably not a substantial claim: may not pass S3 (load-bearing).

We did not assess other factors that influence adoption, such as computer self-efficacy.

“Finally, we did not assess other factors that influence adoption, such as computer self-efficacy, that we did not assess.”

Probably not a substantial claim: may not pass S2 (checkable), S3 (load-bearing). May combine two claims.

Our findings align with, and build on, the work of others.

“Our findings align with, and build on, the work of others.”

Probably not a substantial claim: may not pass S2 (checkable). May combine two claims.

Taken together, these findings highlight the importance of the tool's actionability, explainability, and harm minimization for its implementation and provide a blueprint for researchers interested in developing AI tools for primary care settings.

“Taken together, these findings highlight the importance of the tool’s actionability, explainability, and harm minimization (resulting from bias and workflow disruptions) for its implementation and provide a blueprint for researchers interested in developing AI tools for primary care settings.”

• AssumptionRests on an assumption: End-user input before a prototype is created is important for adoption of an AI tool.

Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

Because participants were recruited from our networks, many were from academic settings and FQHCs; results may differ if the sample were more representative of primary care clinics across the United States.

“First, because we recruited participants from our networks, many of them were from academic settings and FQHCs. Our results may differ if we had a sample that is more representative of primary care clinics across the United States.”