Clinical Utility and Functionality of an Artificial Intelligence–Based App to Predict Mortality in COVID-19: Mixed Methods Analysis
Open accessPublished version of a preprint
Abstract
Background: The artificial neural network (ANN) is an increasingly important tool in the context of solving complex medical classification problems. However, one of the principal challenges in leveraging artificial intelligence technology in the health care setting has been the relative inability to translate models into clinician workflow. Objective: Here we demonstrate the development of a COVID-19 outcome prediction app that utilizes an ANN and assesses its usability in the clinical setting. Methods: Usability assessment was conducted using the app, followed by a semistructured end-user interview. Usability was specified by effectiveness, efficiency, and satisfaction measures. These data were reported with descriptive statistics. The end-user interview data were analyzed using the thematic framework method, which allowed for the development of themes from the interview narratives. In total, 31 National Health Service physicians at a West London teaching hospital, including foundation physicians, senior house officers, registrars, and consultants, were included in this study. Results: All participants were able to complete the assessment, with a mean time to complete separate patient vignettes of 59.35 (SD 10.35) seconds. The mean system usability scale score was 91.94 (SD 8.54), which corresponds to a qualitative rating of “excellent.” The clinicians found the app intuitive and easy to use, with the majority describing its predictions as a useful adjunct to their clinical practice. The main concern was related to the use of the app in isolation rather than in conjunction with other clinical parameters. However, most clinicians speculated that the app could positively reinforce or validate their clinical decision-making. Conclusions: Translating artificial intelligence technologies into the clinical setting remains an important but challenging task. We demonstrate the effectiveness, efficiency, and system usability of a web-based app designed to predict the outcomes of patients with COVID-19 from an ANN.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Usability of and satisfaction with AI tools are generally high.
“researchers are starting to pay attention to end users [35] and are finding that usability of and satisfaction with AI tools are generally high [35-37].”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [37]
The Results (Satisfaction) report a mean usability score of 91.94, rated “excellent,” and the Discussion (Principal Findings) says clinicians found the app intuitive and easy to use. The study assessed one COVID-19 prediction app at a single hospital, so it does not establish that usability and satisfaction are generally high across AI tools. (LLM, conf=0.86)
Experimental — accuracy not yet measuredMost of these studies used quantitative methods and examined tools that have already been developed.
“Most of these studies used quantitative methods and examined tools that have already been developed [35-37].”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [37]
The Methods (Data Analysis; page unavailable) report quantitative usability statistics for an app that participants used, supporting the claim that this study examined a functioning tool. But the same section reports qualitative interview analysis, and Methods (Development of GUI; page unavailable) describes developing and iteratively refining the app. This paper therefore does not support characterizing its approach as solely quantitative or its tool as already developed independently of the study. (LLM, conf=0.84)
Experimental — accuracy not yet measuredOther clinicians have responded favorably to the usability of tools that use AI.
“For example, similar to our results, other clinicians have responded favorably to the usability of tools that use AI [36,37].”
From An Assessment of How Clinicians and Staff Members Use a Diabetes Artificial Intelligence Prediction Tool: Mixed Methods Study (2023) · cites this paper as [37]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Morse, J. M. (2000) · Qualitative Health Research · doi:10.1177/104973200129118183 · Cited by 3,244 · see citing works
Wynants, L., Van Calster, B., Bonten, M. M., Collins, G. S., Debray, T. P., De Vos, M., Haller, M. C., Heinze, G., Moons, K. G., Riley, R. D., et al. (2020) · doi:10.1101/2020.03.24.20041020 · Cited by 3,070 · see citing works
Qualitative Research: Qualitative interviews in medical research
Britten, N. (1995) · BMJ · doi:10.1136/bmj.311.6999.251 · Cited by 1,670 · see citing works
Genetic mechanisms of critical illness in COVID-19
The GenOMICC Investigators, Pairo‐Castineira, E., The ISARIC4C Investigators, Initiative, T. C.-1. H. G., 23andMe Investigators, BRACOVID Investigators, Gen-COVID Investigators, Clohisey, S., Klarić, L., Bretherick, A. D., et al. (2020) · Nature · doi:10.1038/s41586-020-03065-y · Cited by 1,335 · see citing works
Knight, S. R., Ho, A. Y. W., Pius, R., Buchan, I. E., Carson, G., Drake, T. M., Dunning, J. W., Fairfield, C. J., Gamble, C. L., Green, C., et al. (2020) · BMJ · doi:10.1136/bmj.m3339 · Cited by 1,020 · see citing works
Marshall, M. (1996) · Family Practice · doi:10.1093/fampra/13.1.92 · Cited by 769 · see citing works
Shi, Y., Yu, X., Zhao, H., Wang, H., Zhao, R.-H., & Sheng, J. (2020) · Critical Care · doi:10.1186/s13054-020-2833-7 · Cited by 510 · see citing works
Estimation of risk factors for COVID-19 mortality - preliminary resultsPreprint
Caramelo, F. J., Ferreira, N. R., & Oliveiros, B. (2020) · medRxiv · doi:10.1101/2020.02.24.20027268 · Cited by 236 · see citing works
Role of Machine Learning Techniques to Tackle the COVID-19 Crisis: Systematic Review
Syeda, H. B., Syed, M., Sexton, K. W., Syed, S., Begum, S., Syed, F., Prior, F., & Yu, F. (2020) · JMIR Medical Informatics · doi:10.2196/23811 · Cited by 165 · see citing works
Pan, P., Li, Y., Xiao, Y., Han, B., Su, L., Su, M., Li, Y., Zhang, S., Jiang, D., Chen, X., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/23128 · Cited by 128 · see citing works
Charani, E., Castro‐Sánchez, E., Moore, L. S. P., & Holmes, A. H. (2014) · BMC Medicine · doi:10.1186/1741-7015-12-29 · Cited by 114 · see citing works
Self-citationDexamethasone in hospitalised patients with COVID-19: addressing uncertainties
Matthay, M. A., & Thompson, B. (2020) · The Lancet Respiratory Medicine · doi:10.1016/s2213-2600(20)30503-8 · Cited by 112 · see citing works
Clinical Predictive Models for COVID-19: Systematic Study
Schwab, P., Schütte, A. D., Dietz, B., & Bauer, S. (2020) · Journal of Medical Internet Research · doi:10.2196/21439 · Cited by 110 · see citing works
Abdulaal, A., Patel, A. M., Charani, E., Denny, S., Mughal, N., & Moore, L. S. P. (2020) · Journal of Medical Internet Research · doi:10.2196/20259 · Cited by 85 · see citing works
Self-citationThe challenges of deploying artificial intelligence models in a rapidly evolving pandemic
Hu, Y., Jacob, J., Parker, G. J. M., Hawkes, D. J., Hurst, J. R., & Stoyanov, D. V. (2020) · Nature Machine Intelligence · doi:10.1038/s42256-020-0185-2 · Cited by 82 · see citing works
A Human(e) Factor in Clinical Decision Support Systems
Bezemer, T., de Groot, M., Blasse, E., ten Berg, M. J., Kappen, T. H., Bredenoord, A. L., van Solinge, W. W., Hoefer, I. E., & Haitjema, S. (2019) · Journal of Medical Internet Research · doi:10.2196/11732 · Cited by 81 · see citing works
New-onset anosmia and ageusia in adult patients diagnosed with SARS-CoV-2 infection
Patel, A. M., Charani, E., Ariyanayagam, D., Abdulaal, A., Denny, S., Mughal, N., & Moore, L. S. P. (2020) · Clinical Microbiology and Infection · doi:10.1016/j.cmi.2020.05.026 · Cited by 79 · see citing works
Self-citationPaediatric ‘care closer to home’: Stake-holder views and barriers to implementation
Heath, G., Cameron, E., Cummins, C., Greenfield, S., Pattison, H. M., Kelly, D. A., & Redwood, S. (2012) · Health & Place · doi:10.1016/j.healthplace.2012.05.003 · Cited by 69 · see citing works
Quiroz, J. C., Feng, Y., Cheng, Z., Rezazadegan, D., Chen, P.-K., Lin, Q., Qian, L., Liu, X., Berkovsky, S., Coiera, E. W., et al. (2021) · JMIR Medical Informatics · doi:10.2196/24572 · Cited by 52 · see citing works
Miller, K. M., Capan, M., Weldon, D. L. M., Noaiseh, Y., Kowalski, R. L., Kraft, R., Schwartz, S., Weintraub, W. S., & Arnold, R. C. (2018) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2018.05.008 · Cited by 50 · see citing works
Snapshots and snippets: General practitioners’ reflections on professional space
Rapport, F., Doel, M. A., & Elwyn, G. J. (2006) · Health & Place · doi:10.1016/j.healthplace.2006.07.005 · Cited by 47 · see citing works
British Medical Journal Publishing Group (2020) · BMJ · doi:10.1136/bmj.m4334 · Cited by 44 · see citing works
Abdulaal, A., Patel, A. M., Charani, E., Denny, S., Alqahtani, S. A., Davies, G., Mughal, N., & Moore, L. S. P. (2020) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01316-6 · Cited by 39 · see citing works
Self-citationDomínguez-Olmedo, J. L., Gragera-Martínez, Á., Vázquez, J. M., & Pachón, V. (2021) · Journal of Medical Internet Research · doi:10.2196/26211 · Cited by 38 · see citing works
Peiffer‐Smadja, N., Poda, A., Ouédraogo, A., Guiard–Schmid, J., Delory, T., Bel, J. L., Bouvet, É., Lariven, S., Jeanmougin, P., Ahmad, R., et al. (2020) · Journal of Medical Internet Research · doi:10.2196/17940 · Cited by 34 · see citing works
Jung, S. Y., Hwang, H. Y., Lee, K., Lee, H., Kim, E., Kim, M.-Y., & Cho, I. Y. (2020) · JMIR Medical Informatics · doi:10.2196/18758 · Cited by 26 · see citing works
Patrício, A., Costa, R. S., & Henriques, R. (2021) · Journal of Medical Internet Research · doi:10.2196/26075 · Cited by 16 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [21]Creswell JW Poth CN Qualitative inquiry and research design: Choosing among five approaches Internet 2016 Thousand Oaks, CA Sage Publications
Cited by
Sorted by citations
Wang, L., Zhang, Z., Wang, D., Cao, W., Zhou, X., Zhang, P., Liu, J., Fan, X., & Tian, F. (2023) · Frontiers in Computer Science · doi:10.3389/fcomp.2023.1187299 · Cited by 132 · see citing works
Evans, R. P., Bryant, L. D., Russell, G., & Absolom, K. L. (2024) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2024.105342 · Cited by 38 · see citing works
Quality of interaction between clinicians and artificial intelligence systems. A systematic review
Perivolaris, A., Adams-McGavin, C., Madan, Y., Kishibe, T., Antoniou, T., Mamdani, M., & Jung, J. J. (2024) · Future Healthcare Journal · doi:10.1016/j.fhj.2024.100172 · Cited by 38 · see citing works
Parsons, C. S., Zuiderwijk, A., Orchard, N. A., Oosterhoff, J. H. F., & de Reuver, M. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03237-8 · Cited by 32 · see citing works
Okeibunor, J., Jaca, A., Iwu, C. J., Idemili-Aronu, N., Ba, H., Zantsi, Z. P., Ndlambe, A. M., Mavundza, E. J., Muneene, D., Wiysonge, C. S., et al. (2023) · Frontiers in Public Health · doi:10.3389/fpubh.2023.1102185 · Cited by 32 · see citing works
Liaw, W. R., Ramos Silva, Y., Soltero, E. G., Krist, A., & Stotts, A. L. (2023) · JMIR AI · doi:10.2196/45032 · Cited by 14 · see citing works
“For example, similar to our results, other clinicians have responded favorably to the usability of tools that use AI [36,37].”
BackgroundResultCasano, N., Santini, S. J., Vittorini, P., Sinatti, G., Carducci, P., Mastroianni, C. M., Ciardi, M. R., Pasculli, P., Petrucci, E., Marinangeli, F., et al. (2023) · Berichte aus der medizinischen Informatik und Bioinformatik/Journal of integrative bioinformatics · doi:10.1515/jib-2022-0047 · Cited by 11 · see citing works
Peng, H., Hu, C., Deng, W., Huang, L., Zhang, Y., Luo, B., Wang, X., Long, X., & Huang, X. (2022) · BMC Pulmonary Medicine · doi:10.1186/s12890-022-01986-0 · Cited by 9 · see citing works
Scoping Review on Lead User Engagement Methods in the Development of Telemedicine Products
Bhebhe, M., Grobbelaar, S. S. S., & Herselman, M. E. (2023) · International Conference on Information Science and Systems (ICISS) · doi:10.1145/3625156.3625195 · Cited by 1 · see citing works
What Is the Value of Artificial Intelligence in Radiology?
Alrujaib, M., & Bakheet, D. (2021) · Innovations in Surgery and Interventional Medicine · doi:10.36401/isim-21-03
The innovator’s cookbook: engaging lead users in healthcare product development
Bhebhe, M., & Grobbelaar, S. S. S. (2025) · Management Review Quarterly · doi:10.1007/s11301-025-00505-9
Peng, H., Hu, C., Deng, W., Huang, L., Zhang, Y., Luo, B., Wang, X., Long, X., & Huang, X. (2022) · Figshare · doi:10.6084/m9.figshare.19759620
Evaluating Human factors Design in Clinician-Facing AI Applications: A Scoping Review (Preprint)Preprint
Frank, C. A., Stevens, A., & Engineer, L. D. (2026) · doi:10.2196/preprints.107400
- Authors
- Abdulaal, A., Patel, A., Al-Hindawi, A., Charani, E., Alqahtani, S. A., Davies, G. W., Mughal, N., & Moore, L. S. P.
- Affiliations
- Chelsea and Westminster Hospital NHS Foundation Trust (GB)
- National Institute for Health and Care Research (GB)
- Johns Hopkins University (US)
- Imperial College Healthcare NHS Trust (GB)
- Corresponding author
- Moore, Luke Stephen Prockter
- ORCID
- Venue
- JMIR Formative Research
- Publisher
- JMIR Publications Inc.
- Published
- 2021-07-28
- Volume
- 5
- Issue
- 7
- Pages
- e27992
- ISSN
- 2561-326X (ISSN-L)
- PMC
- PMC8320734
- OpenAlex
- W3169554784
- Cited by
- 13
- References
- 27
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 2 · 2023: 5 · 2022: 2 · 2021: 1
- FWCI
- 0.73
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- JMIR Formative Research (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Spiral (Imperial College London) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- National Institute for Health Research Health Protection Research Unit (HPRU_2012_10047)
- National Institute for Health and Care Research (HPRU-2012-10047)
- Imperial College London (HPRU-2012-10047)
- Public Health England (HPRU-2012–10047)
- Economic and Social Research Council
- NIHR Imperial Biomedical Research Centre
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Health Professions › General Health Professions › Mobile Health and mHealth Applications
- Keywords
- COVID-19 pandemic, mortality prediction, artificial neural networks, clinical decision support, mobile health apps, usability, clinician acceptance, clinical workflow integration, System Usability Scale, clinical decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
