Iterative User Interface Design for Automated Sequential Organ Failure Assessment Score Calculator in Sepsis Detection
Open accessThe DOI matches its Crossref record; 10 of the 10 references it cites were found in a registry or an index.
Abstract
BACKGROUND: The new sepsis definition has increased the need for frequent sequential organ failure assessment (SOFA) score recalculation and the clerical burden of information retrieval makes this score ideal for automated calculation. OBJECTIVE: The aim of this study was to (1) estimate the clerical workload of manual SOFA score calculation through a time-motion analysis and (2) describe a user-centered design process for an electronic medical record (EMR) integrated, automated SOFA score calculator with subsequent usability evaluation study. METHODS: First, we performed a time-motion analysis by recording time-to-task-completion for the manual calculation of 35 baseline and 35 current SOFA scores by 14 internal medicine residents over a 2-month period. Next, we used an agile development process to create a user interface for a previously developed automated SOFA score calculator. The final user interface usability was evaluated by clinician end users with the Computer Systems Usability Questionnaire. RESULTS: The overall mean (standard deviation, SD) time-to-complete manual SOFA score calculation time was 61.6 s (33). Among the 24% (12/50) usability survey respondents, our user-centered user interface design process resulted in >75% favorability of survey items in the domains of system usability, information quality, and interface quality. CONCLUSIONS: Early stakeholder engagement in our agile design process resulted in a user interface for an automated SOFA score calculator that reduced clinician workload and met clinicians' needs at the point of care. Emerging interoperable platforms may facilitate dissemination of similarly useful clinical score calculators and decision support algorithms as "apps." A user-centered design process and usability evaluation should be considered during creation of these tools.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
IBM computer usability satisfaction questionnaires: Psychometric evaluation and instructions for use
Lewis, J. R. (1995) · International Journal of Human-Computer Interaction · doi:10.1080/10447319509526110 · Cited by 2,429
Tu, J. V. (1996) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(96)00002-9 · Cited by 1,818 · cited 1×
“Finally, traditional externally validated clinical scores, like SOFA, may be more generalizable than machine-learning algorithms [50].”
MethodHillestad, R., Bigelow, J., Bower, A., Girosi, F., Meili, R., Scoville, R., & Taylor, R. (2005) · Health Affairs · doi:10.1377/hlthaff.24.5.1103 · Cited by 1,675 · cited 1×
“As electronic medical records (EMRs) have propagated through the US health care system, they have brought both great promise and great problems [1,2].”
BackgroundManaging Clinical Knowledge for Health Care Improvement
Balas, E. A., & Boren, S. A. (2000) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1637943 · Cited by 1,479 · cited 1×
“Additionally, the platform offers a means to reduce the 17-year gap between clinical-knowledge generation and widespread usage [52].”
BackgroundRaith, E. P., Udy, A. A., Bailey, M., McGloughlin, S., MacIsaac, C., Bellomo, R., Pilcher, D. V., & for the Australian and New Zealand Intensive Care Society (ANZICS) Centre for Outcomes and Resource Evaluation (CORE) (2017) · JAMA · doi:10.1001/jama.2016.20328 · Cited by 1,279
Shanafelt, T. D., Dyrbye, L. N., Sinsky, C., Hasan, O., Satele, D., Sloan, J., & West, C. P. (2016) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2016.05.007 · Cited by 1,086 · cited 1×
“One unintended consequence of increasing EMR adoption that has been recently characterized is physician burnout associated with EMR-associated clerical tasks [3].”
BackgroundSMART on FHIR: a standards-based, interoperable apps platform for electronic health records
Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S., & Ramoni, R. B. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv189 · Cited by 829
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451 · cited 1×
“Items 3-5 reflect the concerns several stakeholders expressed about the potential for automation bias with this tool [31].”
BackgroundHealth information technology: fallacies and sober realities
Karsh, B.-T., Weinger, M. B., Abbott, P. A., & Wears, R. L. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.005637 · Cited by 365
A computational approach to early sepsis detection
Calvert, J. S., Price, D. A., Chettipally, U. K., Barton, C. W., Feldman, M. D., Hoffman, J. L., Jay, M., & Das, R. (2016) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2016.05.003 · Cited by 244
Cited by
Mechanisms and treatment of organ failure in sepsis
Lelubre, C., & Vincent, J.-L. (2018) · Nature Reviews Nephrology · doi:10.1038/s41581-018-0005-7 · Cited by 852
Wronikowska, M. W., Malycha, J., Morgan, L. J., Westgate, V., Petrinic, T., Young, J. D., & Watkinson, P. J. (2021) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13582 · Cited by 59
A scoping review of silent trials for medical artificial intelligence
Tikhomirov, L., Semmler, C., Prizant, N., Bhasin, S., Kenyon, G., van der Vegt, A., Erdman, L., Kurian, N. C., Thompson, H., Palmer, L. J., et al. (2026) · Nature Health · doi:10.1038/s44360-025-00048-z · Cited by 17
Wan, Y.-K. J., Wright, M. C., McFarland, M. M., Dishman, D., Nies, M. A., Rush, A., Madaras-Kelly, K., Jeppesen, A., & Del Fiol, G. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad203 · Cited by 11
Thissen, A., Salgado-Baez, E., Fürstenau, D., Delucchi Danhier, R., Meske, C., Kuss, P., Angermair, S., Spies, C., & Balzer, F. (2025) · Journal of Medical Internet Research · doi:10.2196/49254 · Cited by 7
“It is crucial for manufacturers to focus on user-centered design and conduct systematic, iterative usability tests [66,67].”
Metadata
- Authors
- Aakre, C. A., Kitson, J. E., Li, M., & Herasevich, V.
- Venue
- JMIR Human Factors
- Publisher
- JMIR Publications Inc.
- Published
- 2017-05-18
- Volume
- 4
- Issue
- 2
- Pages
- e14
- ISSN
- 2292-9495 (ISSN-L)
- OpenAlex
- W2615528965
- Cited by
- 17
- References
- 10
- FWCI
- 1.42
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.