Cited by 61Electronic Health Records SystemsArtificial Intelligence in Healthcare and EducationTechnology Adoption and User Behaviour

Technology acceptance of a machine learning algorithm predicting delirium in a clinical setting: a mixed-methods study

Open access
Jauk, Stefanie, Kramer, Diether, Avian, Alexander, Berghold, Andrea, Leodolter, Werner, Schulz, Stefan·2021·Journal of Medical Systems

Abstract

Early identification of patients with life-threatening risks such as delirium is crucial in order to initiate preventive actions as quickly as possible. Despite intense research on machine learning for the prediction of clinical outcomes, the acceptance of the integration of such complex models in clinical routine remains unclear. The aim of this study was to evaluate user acceptance of an already implemented machine learning-based application predicting the risk of delirium for in-patients. We applied a mixed methods design to collect opinions and concerns from health care professionals including physicians and nurses who regularly used the application. The evaluation was framed by the Technology Acceptance Model assessing perceived ease of use, perceived usefulness, actual system use and output quality of the application. Questionnaire results from 47 nurses and physicians as well as qualitative results of four expert group meetings rated the overall usefulness of the delirium prediction positively. For healthcare professionals, the visualization and presented information was understandable, the application was easy to use and the additional information for delirium management was appreciated. The application did not increase their workload, but the actual system use was still low during the pilot study. Our study provides insights into the user acceptance of a machine learning-based application supporting delirium management in hospitals. In order to improve quality and safety in healthcare, computerized decision support should predict actionable events and be highly accepted by users.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1007/s10916-021-01727-6

Cites

31 of 31 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Participants commonly reported that XAI tools support task completion.

    “Across these studies participants commonly reported that the XAI tools support task completion [23,26,28,29,31,33].”

    From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [31]

    In Results > Expert group meetings (page unavailable), the expert group said the application supported delirium-risk assessment and reduced screening time. But Results > Technology acceptance questionnaire (page unavailable) reports mixed views among the 47 surveyed users: 36.2% found it useful for their work and 31.9% did not. The paper supports reports of task support, but not the unqualified suggestion that participants commonly reported it. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • Integration with electronic health records was common, particularly in more mature tools.

    “Integration with electronic health records (EHRs) was common, particularly in more mature tools [23,24,26–34,36].”

    From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [31]

    The Study design section describes one delirium-prediction application integrated into a hospital information system, and the Discussion says it uses EHR-based risk prediction. The paper does not establish that EHR integration was common across tools or particularly prevalent in more mature tools; its Conclusion describes this implementation as among the first of its kind. No page numbers are available in our copy. (LLM, conf=0.86)

    Experimental — accuracy not yet measured
  • Participants in several studies described the role of the model's performance in influencing trust.

    “Participants in several studies described the role of the model’s performance in influencing trust [23,25,31–33,36].”

    From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [31]

    In Results > Expert group meetings (page not provided), participants commented on the model’s predictive accuracy, saying it had “almost 100 % accuracy” and that its risk-group assignments “seem correct.” This supports the performance aspect, but the passage does not say that performance influenced their trust; a search for trust found no explicit account of that link. (LLM, conf=0.76)

    Experimental — accuracy not yet measured
  • Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow.

    “Health professionals positively perceived, and were more likely to use, XAI systems that fit into their existing workflow [23–26,30,31,33,35].”

    From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [31]

    The Results (Expert group meetings; page not provided) report positive views of the tool’s ease of use, including its availability in the hospital information system without extra data entry. But the Discussion says some users could not integrate it into their clinical routine and actual use was low. The paper does not establish that professionals whose workflows accommodated the tool were more likely to use it. (LLM, conf=0.85)

    Experimental — accuracy not yet measured
  • A number of studies reported trust could be earned over time as model performance was proven.

    “A number of studies also reported trust could be earned over time as model performance was proven [23,25,31–33,36], for example, an electrophysiologist said “I just think I should see that it confirms our decisions in enough cases - then I would feel comfortable about colleagues leaning on it” [33].”

    From Explainable AI in hospital clinical decision support systems: A scoping review of healthcare professionals’ perspectives (2026) · cites this paper as [31]

    The Conclusion (page unavailable) links the algorithm’s high accuracy with positive acceptance by physicians and nurses, and Results > Expert group meetings reports that experts perceived its accuracy as high. The paper does not report that trust was earned over time as performance was demonstrated. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

1 of the 32 references couldn’t be matched to a known paper.

Cited by

Sorted by citations

Authors
Jauk, S., Kramer, D., Avian, A., Berghold, A., Leodolter, W., & Schulz, S.
Affiliations
Medical University of Graz (AT)
Corresponding author
Jauk, Stefanie
Venue
Journal of Medical Systems
Published
2021-03-01
Volume
45
Issue
4
Pages
48-48
ISSN
  • 0148-5598 (ISSN-L)
  • 1573-689X
OpenAlex
W3135561137
Cited by
59
References
32
Citations by year
2026: 12 · 2025: 14 · 2024: 15 · 2023: 10 · 2022: 7 · 2021: 1
FWCI
7.19
Type
journal article
Language
en
Open access
Locations
  • Journal of Medical Systems (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
Funders
Medical University of Graz
Topics
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Social Sciences › Decision Sciences › Information Systems and Management › Technology Adoption and User Behaviour
Keywords
delirium prediction, machine learning, technology acceptance, technology acceptance model, clinical decision support, hospitalized patients, healthcare workers, perceived usefulness, perceived ease of use, delirium management, inpatient care, user acceptance
MeSH terms
Clinical Decision-Making, Machine Learning, Algorithms, Australia, Delirium, Diagnosis, Differential, Diagnostic Errors, Female, Humans, Male, Middle Aged, Pilot Projects, Psychiatric Status Rating Scales, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review