Participatory design of probability-based decision support tools for in-hospital nurses
Open accessAbstract
OBJECTIVE: To describe nurses' preferences for the design of a probability-based clinical decision support (PB-CDS) tool for in-hospital clinical deterioration. METHODS: A convenience sample of bedside nurses, charge nurses, and rapid response nurses (n = 20) from adult and pediatric hospitals completed participatory design sessions with researchers in a simulation laboratory to elicit preferred design considerations for a PB-CDS tool. Following theme-based content analysis, we shared findings with user interface designers and created a low-fidelity prototype. RESULTS: Three major themes and several considerations for design elements of a PB-CDS tool surfaced from end users. Themes focused on "painting a picture" of the patient condition over time, promoting empowerment, and aligning probability information with what a nurse already believes about the patient. The most notable design element consideration included visualizing a temporal trend of the predicted probability of the outcome along with user-selected overlapping depictions of vital signs, laboratory values, and outcome-related treatments and interventions. Participants expressed that the prototype adequately operationalized requests from the design sessions. CONCLUSIONS: Participatory design served as a valuable method in taking the first step toward developing PB-CDS tools for nurses. This information about preferred design elements of tools that support, rather than interrupt, nurses' cognitive workflows can benefit future studies in this field as well as nurses' practice.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Medical experts may be unable to understand explained risk factors and estimated probabilistic explanations.
“Also, there is a possibility that the medical experts may be unable to understand the explained risk factors and estimated probabilistic explanations [20].”
From Explainable, trustworthy, and ethical machine learning for healthcare: A survey (2022) · cites this paper as [20]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
User Acceptance of Information Technology: Toward A Unified View1
Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003) · MIS Quarterly · doi:10.2307/30036540 · Cited by 45,035 · see citing works
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · see citing works
Heart Disease and Stroke Statistics—2014 Update
Go, A. S., Mozaffarian, D., Roger, V. L., Benjamin, E. J., Berry, J. D., Blaha, M. J., Dai, S., Ford, E. S., Fox, C. S., Franco, S., et al. (2013) · Circulation · doi:10.1161/01.cir.0000441139.02102.80 · Cited by 5,207 · see citing works
Yeh, M. (2014) · Journal of Nursing Research · doi:10.1097/jnr.0000000000000065 · Cited by 3,836 · see citing works
Work system design for patient safety: the SEIPS model
Carayon, P., Schoofs Hundt, A., Karsh, B.-T., Gurses, A. P., Alvarado, C. J., Smith, M., & Flatley Brennan, P. (2006) · Quality in Health Care · doi:10.1136/qshc.2005.015842 · Cited by 1,795 · see citing works
Risk Prediction Models for Hospital Readmission
Kansagara, D. L., Englander, H. L., Salanitro, A. H., Kagen, D. I., Theobald, C. N., Freeman, M., & Kripalani, S. B. (2011) · JAMA · doi:10.1001/jama.2011.1515 · Cited by 1,712 · see citing works
Big Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235 · see citing works
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200 · see citing works
Collins, G. S., de Groot, J. A., Dutton, S., Omar, O., Shanyinde, M., Tajar, A., Voysey, M., Wharton, R., Yu, L.-M., Moons, K. G., et al. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-40 · Cited by 729 · see citing works
Jaspers, M. W. M. (2009) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.10.002 · Cited by 649 · see citing works
Jaspers, M. W. M., Smeulers, M., Vermeulen, H., & Peute, L. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000094 · Cited by 525 · see citing works
Health Information Technology: An Updated Systematic Review With a Focus on Meaningful Use
Jones, S. S., Rudin, R. S., Perry, T., & Shekelle, P. G. (2014) · Annals of Internal Medicine · doi:10.7326/m13-1531 · Cited by 475 · see citing works
Incidence of treated cardiac arrest in hospitalized patients in the United States*
Merchant, R. M., Yang, L., Becker, L. B., Berg, R. A., Nadkarni, V., Nichol, G., Carr, B. G., Mitra, N., Bradley, S. M., Abella, B. S., et al. (2011) · Critical Care Medicine · doi:10.1097/ccm.0b013e3182257459 · Cited by 429 · see citing works
Hodgetts, T. J., Kenward, G., Vlackonikolis, I., Payne, S., Nicolas Castle, B., Crouch, R., Ineson, N., & Shaikh, L. A. H. A. (2002) · Resuscitation · doi:10.1016/s0300-9572(02)00098-9 · Cited by 413 · see citing works
Participatory IT Design: Designing for Business and Workplace Realities
Bødker, K., Kensing, F., & Simonsen, J. (2004) · Cited by 346 · see citing works
Li, A. C., Kannry, J. L., Kushniruk, A. W., Chrimes, D., McGinn, T. G., Edonyabo, D., & Mann, D. M. (2012) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2012.02.009 · Cited by 197 · see citing works
Collins, G. S., Omar, O., Shanyinde, M., & Yu, L.-M. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.06.020 · Cited by 192 · see citing works
Escobar, G. J., LaGuardia, J. C., Turk, B. J., Ragins, A. I., Kipnis, P., & Draper, D. (2012) · Journal of Hospital Medicine · doi:10.1002/jhm.1929 · Cited by 185 · see citing works
Implementing Electronic Health Care Predictive Analytics: Considerations And Challenges
Amarasingham, R., Patzer, R. E., Huesch, M. D., Nguyen, N. Q., & Xie, B. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0352 · Cited by 180 · see citing works
Churpek, M. M., Yuen, T. C., Park, S. Y., Gibbons, R. D., & Edelson, D. P. (2013) · Critical Care Medicine · doi:10.1097/ccm.0000000000000038 · Cited by 138 · see citing works
Ettema, R. G. A., Peelen, L. M., Schuurmans, M. J., Nierich, A. P., Kalkman, C. J., & Moons, K. G. M. (2010) · Circulation · doi:10.1161/circulationaha.109.926808 · Cited by 130 · see citing works
Bailey, T. C., Chen, Y., Mao, Y., Lu, C., Hackmann, G., Micek, S. T., Heard, K. M., Faulkner, K. M., & Kollef, M. H. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2009 · Cited by 121 · see citing works
Finlay, G. D., Rothman, M. J., & Smith, R. A. (2013) · Journal of Hospital Medicine · doi:10.1002/jhm.2132 · Cited by 119 · see citing works
Thursky, K., & Mahemoff, M. (2006) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2006.07.011 · Cited by 95 · see citing works
Kollef, M. H., Chen, Y., Heard, K. M., LaRossa, G. N., Lu, C., Martin, N. R., Martin, N., Micek, S. T., & Bailey, T. C. (2014) · Journal of Hospital Medicine · doi:10.1002/jhm.2193 · Cited by 91 · see citing works
Integrative review of clinical decision support for registered nurses in acute care settings
Dunn Lopez, K., Gephart, S. M., Raszewski, R., Sousa, V., Shehorn, L. E., & Abraham, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw084 · Cited by 83 · see citing works
Clinical information displays to improve ICU outcomes
Effken, J. A., Loeb, R. G., KANG, Y., & Lin, Z. (2008) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2008.05.004 · Cited by 73 · see citing works
Müller‐Staub, M., de Graaf-Waar, H., & Paans, W. (2016) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000277 · Cited by 72 · see citing works
Álvarez, C., Clark, C. A., Zhang, S., Halm, E. A., Shannon, J. J., Girod, C. E., Cooper, L., & Amarasingham, R. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-28 · Cited by 66 · see citing works
Automated detection of physiologic deterioration in hospitalized patients
Evans, R. S., Kuttler, K. G., Simpson, K. J., HOWE, S. E., Crossno, P. F., Johnson, K. V., Schreiner, M. N., Lloyd, J. F., Tettelbach, W. H., Keddington, R. K., et al. (2014) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2014-002816 · Cited by 58 · see citing works
The significance of cognitive modeling in building healthcare interfaces
JOHNSON, C., & Turley, J. P. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.06.003 · Cited by 53 · see citing works
A Clinical Deterioration Prediction Tool for Internal Medicine Patients
Kirkland, L. L., Malinchoc, M., O’Byrne, M. M., et al. (2012) · American Journal of Medical Quality · doi:10.1177/1062860612450459 · Cited by 38 · see citing works
Fillmore, C. L., Bray, B. E., & Kawamoto, K. (2013) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-13-135 · Cited by 37 · see citing works
The impact of telemetry on survival of in-hospital cardiac arrests in non-critical care patients
Cleverley, K., Mousavi, N., Stronger, L., et al. (2013) · Resuscitation · doi:10.1016/j.resuscitation.2013.01.038 · Cited by 37 · see citing works
Medical and nursing diagnoses: a critical comparison
Chiffi, D., & Zanotti, R. (2014) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12146 · Cited by 36 · see citing works
Dekker, S. W. A. (2011) · Applied Ergonomics · doi:10.1016/j.apergo.2011.07.003 · Cited by 34 · see citing works
Reeder, B., Hills, R., Turner, A. M., & Demiris, G. (2013) · Public Health Nursing · doi:10.1111/phn.12081 · Cited by 34 · see citing works
"Smallball" evaluation: a prescription for studying community-based information interventions.
Friedman, C. P. (2005) · PubMed · Cited by 23 · see citing works
SOUSA, V. E. C., LOPEZ, K. D., FEBRETTI, A., STIFTER, J., YAO, Y., JOHNSON, A., WILKIE, D. J., & KEENAN, G. M. (2015) · CIN: Computers, Informatics, Nursing · doi:10.1097/cin.0000000000000185 · Cited by 18 · see citing works
Jeffery, A. D. (2015) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000154 · Cited by 14 · see citing works
Self-citationEvaluating Clinical Decision Support Rules as an Intervention in Clinician Workflows With Technology
Brokel, J. M., Schwichtenberg, T., Wakefield, D. S., et al. (2010) · CIN Computers Informatics Nursing · doi:10.1097/ncn.0b013e3181f9dbb1 · Cited by 11 · see citing works
Exploring the Person-Centeredness of an Innovative E-Supportive System Aimed at Person-Centered Care
Ventura, F. I. Q. S., Koinberg, I., Sawatzky, R. G., Karlsson, P., & Öhlén, J. (2016) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000225 · Cited by 9 · see citing works
Multivariable Risk Prediction Models
Collins, P. G. S., & Manach, Y. L. (2013) · Anesthesiology · doi:10.1097/aln.0b013e31828e13e9 · Cited by 4 · see citing works
Flagged first, then by citations
Research works printed but not linked: 3
- Amarasingham R, Patzer RE, Huesch M, Nguyen NQ, Xie B. Implement- ing electronic health care predictive analytics: considerations and chal- lenges. Health Aff (Millwood). 2014;33(7):1148-54.
- Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of infor- mation technology: Toward a unified view. Manag Inform Sys Quart. 2003;27(3):425-78.
- Miles MB, Huberman AM, Saldana J. Qualitative Daya Analysis: A Meth- ods Sourcebook. 3rd ed. Los Angeles: SAGE Publications; 2014.
Cited by
Sorted by citations
Explainable, trustworthy, and ethical machine learning for healthcare: A survey
Rasheed, K., Qayyum, A., Ghaly, M., Al-Fuqaha, A., Razi, A., & Qadir, J. (2022) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2022.106043 · Cited by 372 · see citing works
- Authors
- Jeffery, A. D., Novak, L. L., Kennedy, B., Dietrich, M. S., & Mion, L. C.
- Affiliations
- Vanderbilt University (US)
- The Ohio State University (US)
- Corresponding author
- Jeffery, Alvin D
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-06-19
- Volume
- 24
- Issue
- 6
- Pages
- 1102-1110
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6580937
- OpenAlex
- W2684733610
- Cited by
- 46
- References
- 43
- Citations by year
- 2026: 1 · 2025: 3 · 2024: 6 · 2023: 5 · 2022: 5 · 2021: 5 · 2020: 10 · 2019: 6 · 2018: 5
- FWCI
- 7.88
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- public-domain
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, public-domain)
- PubMed Central (publishedVersion, public-domain)
- PubMed (publishedVersion)
- AMIA
- Funders
- Vanderbilt University
- National Institutes of Health (UL1TR000445)
- Agency for Healthcare Research and Quality
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Surgery › Healthcare Technology and Patient Monitoring
- Health Sciences › Medicine › Family Practice › Clinical Reasoning and Diagnostic Skills
- Keywords
- clinical deterioration, participatory design, nurse preferences, rapid response, temporal data visualization, nurse empowerment, low-fidelity prototypes, bedside nurses, children's hospitals
- MeSH terms
- Clinical Decision-Making, Decision Support Techniques, Humans, Nursing Staff, Hospital, Probability, User-Computer Interface, Models, Statistical, Nursing Methodology Research, Patient Simulation, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
