Principles for Real-World Implementation of Bedside Predictive Analytics Monitoring
Open accessAbstract
A new development in the practice of medicine is Artificial Intelligence-based predictive analytics that forewarn clinicians of future deterioration of their patients. This proactive opportunity, though, is different from the reactive stance that clinicians traditionally take. Implementing these tools requires new ideas about how to educate clinician users to facilitate trust and adoption and to promote sustained use. Our real-world hospital experience implementing a predictive analytics monitoring system that uses electronic health record and continuous monitoring data has taught us principles that we believe to be applicable to the implementation of other such analytics systems within the health care environment. These principles are mentioned below: • To promote trust, the science must be understandable. • To enhance uptake, the workflow should not be impacted greatly. • To maximize buy-in, engagement at all levels is important. • To ensure relevance, the education must be tailored to the clinical role and hospital culture. • To lead to clinical action, the information must integrate into clinical care. • To promote sustainability, there should be periodic support interactions after formal implementation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Clinicians expressed the need to understand both the scientific and clinical bases of the recommendations provided by the AI to confidently validate and apply the decision.
“Clinicians expressed the need to understand both the scientific and clinical bases of the recommendations provided by the AI to confidently validate and apply the decision [19,20,36-38].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [38]
The Framework (page not provided) says clinicians need to understand how the predictive tool works and that examining its scientific basis and patients’ clinical courses helped build confidence. It describes AI-generated risk scores rather than recommendations, and does not say clinicians needed this understanding to validate and apply a particular decision. (LLM, conf=0.85)
Experimental — accuracy not yet measuredSuccessful adoption was achieved by publishing evidence on the algorithm's underpinnings and providing clinicians with details on how data elements interacted within the algorithm to produce the predictive output.
“Moorman [38] reported that successful adoption was achieved by publishing evidence on the algorithm’s underpinnings and providing clinicians with details on how data elements interacted within the algorithm to produce the predictive output.”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [38]
The Framework section says the authors presented an existing published evidence base to clinicians and used patient-focused sessions to show how data elements produced a risk score, helping develop confidence in the tool. The Conclusion identifies understandable science as an ingredient of successful implementation. The source does not say that publishing evidence was the adoption strategy; it says the authors presented evidence that had already been published. (LLM, conf=0.84)
Experimental — accuracy not yet measuredDissatisfaction was reported if recommendations were too similar, inappropriate, or not useful.
“This also included dissatisfaction if recommendations were too similar, inappropriate, or not useful [35,37,38,40,56].”
From Barriers to and facilitators of artificial intelligence adoption in health care: scoping review (2024) · cites this paper as [38]
The Framework and Principles for Implementing Predictive Analytics Monitoring sections discuss clinicians’ skepticism about a predictive score and how to make the tool relevant to their work. They do not report dissatisfaction because recommendations were too similar, inappropriate, or not useful (searched for those terms and related descriptions of dissatisfaction). No page numbers were available in our copy. (LLM, conf=0.76)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Proctor, E., Silmere, H., Raghavan, R., Hovmand, P., Aarons, G., Bunger, A., Griffey, R., & Hensley, M. (2011) · Administration and Policy in Mental Health and Mental Health Services Research · doi:10.1007/s10488-010-0319-7 · Cited by 8,721 · see citing works
Trust in Automation: Designing for Appropriate Reliance
Lee, J. D., & See, K. A. (2004) · Human Factors: The Journal of the Human Factors and Ergonomics Society · doi:10.1518/hfes.46.1.50_30392 · Cited by 6,605 · cited 1× · see citing works
“11,13,16,17 We presented the published evidence base 6,18 – 22 to provide transparency into the algorithms ’ underpinnings and to emphasize the strengths of the scienti fi c foundation.”
BackgroundA surgical safety checklist to reduce morbidity and mortality in a global population
Haynes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A.-H. S., Dellinger, E. P., Herbosa, T., Joseph, S., Kibatala, P. L., Lapitan, M. C. M., et al. (2009) · New England Journal of Medicine · doi:10.1056/nejmsa0810119 · Cited by 5,688 · see citing works
From best evidence to best practice: effective implementation of change in patients' care
Grol, R., & Grimshaw, J. (2003) · The Lancet · doi:10.1016/s0140-6736(03)14546-1 · Cited by 4,531 · see citing works
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · see citing works
Leonard, M. (2004) · BMJ Quality & Safety · doi:10.1136/qshc.2004.010033 · Cited by 2,346 · see citing works
APACHE—acute physiology and chronic health evaluation: a physiologically based classification system
Knaus, W. A., Zimmerman, J. E., Wagner, D. P., Draper, E. A., & Lawrence, D. E. (1981) · Critical Care Medicine · doi:10.1097/00003246-198108000-00008 · Cited by 2,190 · see citing works
Clinical decision support in the era of artificial intelligence
Shortliffe, E. H., & Sepúlveda, M. J. (2018) · JAMA · doi:10.1001/jama.2018.17163 · Cited by 920 · see citing works
What Do We Know about Health Care Team Effectiveness? A Review of the Literature
Lemieux-Charles, L., & McGuire, W. L. (2006) · Medical Care Research and Review · doi:10.1177/1077558706287003 · Cited by 844 · see citing works
McLean, S. F. (2016) · Journal of Medical Education and Curricular Development · doi:10.4137/jmecd.s20377 · Cited by 759 · see citing works
Improving communication in the ICU using daily goals
Pronovost, P. J., Berenholtz, S. M., Dorman, T., Lipsett, P. A., Simmonds, T., & Haraden, C. (2003) · Journal of Critical Care · doi:10.1053/jcrc.2003.50008 · Cited by 660 · see citing works
Impact of the communication and patient hand-off tool SBAR on patient safety: a systematic review
Müller, M., Jürgens, J., Redaèlli, M., Klingberg, K., Hautz, W. E., & Stock, S. (2018) · BMJ Open · doi:10.1136/bmjopen-2018-022202 · Cited by 507 · see citing works
Interprofessional education: tips for design and implementation
van Diggele, C., Roberts, C., Burgess, A. W., & Mellis, C. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02286-z · Cited by 410 · see citing works
Moorman, J. R., Carlo, W. A., Kattwinkel, J., Schelonka, R. L., Porcelli, P. J., Navarrete, C. T., Bancalari, E., Aschner, J. L., Walker, M. W., Perez, J. A. R., et al. (2011) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2011.06.044 · Cited by 369 · see citing works
Starmer, A. J., Sectish, T. C., Simon, D. W., Keohane, C., McSweeney, M. E., Chung, E. Y., Yoon, C. S., Lipsitz, S. R., Wassner, A. J., Harper, M. B., et al. (2013) · JAMA · doi:10.1001/jama.2013.281961 · Cited by 337 · see citing works
Development of a framework for knowledge translation: understanding user context
Jacobson, N. A., Butterill, D., & Goering, P. N. (2003) · Journal of Health Services Research & Policy · doi:10.1258/135581903321466067 · Cited by 332 · see citing works
The right time to learn: Mechanisms and optimization of spaced learning
Smolen, P., Zhang, Y., & Byrne, J. H. (2016) · Nature Reviews Neuroscience · doi:10.1038/nrn.2015.18 · Cited by 326 · see citing works
Developing a team performance framework for the intensive care unit*
Reader, T. W., Flin, R. H., Mearns, K. J., & Cuthbertson, B. H. (2009) · Critical Care Medicine · doi:10.1097/ccm.0b013e31819f0451 · Cited by 295 · see citing works
Laï, M.-C., Brian, M., & Mamzer, M.-F. (2020) · Journal of Translational Medicine · doi:10.1186/s12967-019-02204-y · Cited by 294 · see citing works
Jain, M. (2006) · BMJ Quality & Safety · doi:10.1136/qshc.2005.016576 · Cited by 270 · see citing works
Reducing Medical Errors and Adverse Events
Pham, J. C., Aswani, M. S., Rosen, M. A., Lee, H., Huddle, M. G., Weeks, K. R., & Pronovost, P. J. (2011) · Annual Review of Medicine · doi:10.1146/annurev-med-061410-121352 · Cited by 201 · see citing works
Beaulieu‐Jones, B. K., Yuan, W., Brat, G. A., Beam, A. L., Weber, G. M., Ruffin, M., & Kohane, I. S. (2021) · npj Digital Medicine · doi:10.1038/s41746-021-00426-3 · Cited by 157 · see citing works
Septicemia mortality reduction in neonates in a heart rate characteristics monitoring trial
Fairchild, K. D., Schelonka, R. L., Kaufman, D. A., et al. (2013) · Pediatric Research · doi:10.1038/pr.2013.136 · Cited by 151 · see citing works
Moss, T. J., Lake, D. E., Calland, J. F., Enfield, K. B., Delos, J. B., Fairchild, K. D., & Moorman, J. R. (2016) · Critical Care Medicine · doi:10.1097/ccm.0000000000001738 · Cited by 98 · see citing works
Barriers and facilitators perceived by physicians when using prediction models in practice
Kappen, T. H., van Loon, K., Kappen, M. A. M., van Wolfswinkel, L., Vergouwe, Y., van Klei, W. A., Moons, K. G. M., & Kalkman, C. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.09.008 · Cited by 76 · see citing works
Adaptation of Clinical Prediction Models for Application in Local Settings
Kappen, T. H., Vergouwe, Y., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., & Moons, K. G. M. (2012) · Medical Decision Making · doi:10.1177/0272989x12439755 · Cited by 70 · see citing works
Ruminski, C., Clark, M., Lake, D. E., Kitzmiller, R. R., Keim‐Malpass, J., Robertson, M. P., Simons, T. R., Moorman, J. R., & Calland, J. F. (2018) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-018-0194-4 · Cited by 54 · see citing works
Early Detection of In-Patient Deterioration: One Prediction Model Does Not Fit All
Blackwell, J., Keim‐Malpass, J., Clark, M., Kowalski, R. L., Najjar, S. N., Bourque, J. M., Lake, D. E., & Moorman, J. R. (2020) · Critical Care Explorations · doi:10.1097/cce.0000000000000116 · Cited by 51 · see citing works
Precision Health Analytics With Predictive Analytics and Implementation Research
Pearson, T. A., Califf, R. M., Roper, R., Engelgau, M. M., Khoury, M. J., Alcántara, C., Blakely, C. H., Boyce, C. A., Brown, M. K., Croxton, T. L., et al. (2020) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2020.05.043 · Cited by 49 · see citing works
Moss, T. J., Clark, M., Calland, J. F., Enfield, K. B., Voss, J. D., Lake, D. E., & Moorman, J. R. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181448 · Cited by 47 · see citing works
Predicting the need for urgent intubation in a surgical/trauma intensive care unit
Politano, A. D., Riccio, L. M., Lake, D. E., Rusin, C. G., Guin, L. E., Josef, C. S., Clark, M., Sawyer, R. G., Moorman, J. R., & Calland, J. F. (2013) · Surgery · doi:10.1016/j.surg.2013.05.025 · Cited by 45 · see citing works
Advancing Continuous Predictive Analytics Monitoring
Keim‐Malpass, J., Kitzmiller, R. R., Skeeles-Worley, A. D., Lindberg, C. C., Clark, M., Tai, R. H., Calland, J. F., Sullivan, K., Moorman, J. R., & Anderson, R. A. (2018) · Critical Care Nursing Clinics of North America · doi:10.1016/j.cnc.2018.02.009 · Cited by 41 · see citing works
Attitudes and beliefs about the surgical safety checklist: Just another tick box?
Dharampal, N., Cameron, C., Dixon, E., Ghali, W. A., & Quan, M. L. (2016) · Canadian Journal of Surgery · doi:10.1503/cjs.002016 · Cited by 40 · see citing works
Kastner, M., Li, J. J., Lottridge, D. M., Marquez, C., Newton, D. C., & Straus, S. E. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-40 · Cited by 33 · see citing works
Spaeder, M. C., Moorman, J. R., Tran, C., et al. (2019) · Pediatric Research · doi:10.1038/s41390-019-0518-1 · Cited by 33 · see citing works
Korach, Z. T., Cato, K. D., Collins, S. A., Kang, M. J., Knaplund, C., Dykes, P. C., Wang, L., Schnock, K. O., Garcia, J. P., Jia, H., et al. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-3401814 · Cited by 32 · see citing works
Linnander, E. L., McNatt, Z., Boehmer, K. R., et al. (2020) · BMJ Quality & Safety · doi:10.1136/bmjqs-2019-010734 · Cited by 30 · see citing works
Holder, A. L., & Clermont, G. (2014) · Critical Care Clinics · doi:10.1016/j.ccc.2014.08.007 · Cited by 25 · see citing works
Kitzmiller, R., Vaughan, A., Skeeles-Worley, A., Keim-Malpass, J., Yap, T., Lindberg, C., Kennerly, S., Mitchell, C., Tai, R., Sullivan, B., et al. (2019) · Applied Clinical Informatics · doi:10.1055/s-0039-1688478 · Cited by 25 · see citing works
Baig, M. M., GholamHosseini, H., Gutiérrez, J., Ullah, E., & Lindén, M. (2021) · Applied Clinical Informatics · doi:10.1055/s-0040-1719043 · Cited by 22 · see citing works
Bridging the Quality Chasm: Interprofessional Teams to the Rescue?
Weinstein, R. S., Brandt, B. F., GILBERT, J. H. V., et al. (2013) · The American Journal of Medicine · doi:10.1016/j.amjmed.2012.10.014 · Cited by 19 · see citing works
Schelonka, R. L., Carlo, W. A., Bauer, C. R., et al. (2020) · The Journal of Pediatrics · doi:10.1016/j.jpeds.2019.12.066 · Cited by 18 · see citing works
Understanding end-user support for health information technology: a theoretical framework
Shachak, A., Barnsley, J. M., Tu, K., et al. (2011) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v19i3.810 · Cited by 17 · see citing works
Davis, J. P., Wessells, D. A., & Moorman, J. R. (2020) · Critical Care Explorations · doi:10.1097/cce.0000000000000294 · Cited by 17 · see citing works
Learning and Consolidation as Re-representation: Revising the Meaning of Memory
Wiggins, G. A., & Sanjekdar, A. (2019) · Frontiers in Psychology · doi:10.3389/fpsyg.2019.00802 · Cited by 16 · see citing works
Dynamic data in the ED predict requirement for ICU transfer following acute care admission
Glass, G. F., Hartka, T., Keim‐Malpass, J., et al. (2020) · Journal of Clinical Monitoring and Computing · doi:10.1007/s10877-020-00500-3 · Cited by 13 · see citing works
Lund, O., Andersen, B. S., & Christensen, M. K. (2016) · International Journal of Medical Education · doi:10.5116/ijme.57d5.5693 · Cited by 10 · see citing works
Physician leadership and health information exchange: literature review
Heath, M. L., & Porter, T. H. (2019) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2019-100080 · Cited by 9 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Petersson, L., Larsson, I., Nygren, J. M., Nilsen, P., Neher, M., Reed, J. E., Tyskbo, D., & Svedberg, P. (2022) · BMC Health Services Research · doi:10.1186/s12913-022-08215-8 · Cited by 434 · see citing works
Barriers to and facilitators of artificial intelligence adoption in health care: scoping review
Hassan, M., Kushniruk, A., & Borycki, E. (2024) · JMIR Human Factors · doi:10.2196/48633 · Cited by 324 · see citing works
Artificial Intelligence Applications in Health Care Practice: Scoping Review
Sharma, M., Savage, C., Nair, M., Larsson, I., Svedberg, P., & Nygren, J. M. (2022) · Journal of Medical Internet Research · doi:10.2196/40238 · Cited by 198 · see citing works
Nair, M., Svedberg, P., Larsson, I., & Nygren, J. M. (2024) · PLOS ONE · doi:10.1371/journal.pone.0305949 · Cited by 179 · see citing works
- Authors
- Moorman, L. P.
- Affiliations
- Clinical Implementation Specialist, Advanced Medical Predictive Devices, Diagnostics, and Displays (AMP3D), Charlottesville, Virginia, United States
- Corresponding author
- Moorman, Liza Prudente
- Venue
- Applied Clinical Informatics
- Publisher
- Georg Thieme Verlag KG
- Published
- 2021-08
- Volume
- 12
- Issue
- 04
- Pages
- 888-896
- ISSN
- 1869-0327 (ISSN-L)
- PMC
- PMC8458037
- OpenAlex
- W3200845801
- Cited by
- 26
- References
- 48
- Citations by year
- 2026: 3 · 2025: 6 · 2024: 6 · 2023: 1 · 2022: 9 · 2021: 1
- FWCI
- 3.70
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Applied Clinical Informatics (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, other-oa)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- Funders
- University of Virginia
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- predictive analytics, clinical deterioration prediction, artificial intelligence, implementation principles, clinician education, trust in AI, technology adoption, workflow integration, electronic health records, hospital culture, implementation sustainability, clinical decision support
- MeSH terms
- Artificial Intelligence, Hospitals, Humans, Medicine, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
