Designing Clinical Decision Support Systems (CDSS)—A User-Centered Lens of the Design Characteristics, Challenges, and Implications: Systematic Review
Open accessThe DOI matches its Crossref record; 102 of the 102 references it cites were found in a registry or an index.
Abstract
Background: Clinical decision support systems (CDSS) have the potential to play a crucial role in enhancing health care quality by providing evidence-based information to clinicians at the point of care. Despite their increasing popularity, there is a lack of comprehensive research exploring their design characterization and trends. This limits our understanding and ability to optimize their functionality, usability, and adoption in health care settings. Objective: This systematic review examined the design characteristics of CDSS from a user-centered perspective, focusing on user-centered design (UCD), user experience (UX), and usability, to identify related design challenges and provide insights into the implications for future design of CDSS. Methods: This review followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) recommendations and used a grounded theory analytical approach to guide the conduct, data analysis, and synthesis. A search of 4 major electronic databases (PubMed, Web of Science, Scopus, and IEEE Xplore) was conducted for papers published between 2013 and 2023, using predefined design-focused keywords (design, UX, implementation, evaluation, usability, and architecture). Papers were included if they focused on a designed CDSS for a health condition and discussed the design and UX aspects (eg, design approach, architecture, or integration). Papers were excluded if they solely covered technical implementation or architecture (eg, machine learning methods) or were editorials, reviews, books, conference abstracts, or study protocols. Results: Out of 1905 initially identified papers, 40 passed screening and eligibility checks for a full review and analysis. Analysis of the studies revealed that UCD is the most widely adopted approach for designing CDSS, with all design processes incorporating functional or usability evaluation mechanisms. The CDSS reported were mainly clinician-facing and mostly stand-alone systems, with their design lacking consideration for integration with existing clinical information systems and workflows. Through a UCD lens, four key categories of challenges relevant to CDSS design were identified: (1) usability and UX, (2) validity and reliability, (3) data quality and assurance, and (4) design and integration complexities. Notably, a subset of studies incorporating Explainable artificial intelligence highlighted its emerging role in addressing key challenges related to validity and reliability by fostering explainability, transparency, and trust in CDSS recommendations, while also supporting collaborative validation with users. Conclusions: While CDSS show promise in enhancing health care delivery, identified challenges have implications for their future design, efficacy, and utilization. Adopting pragmatic UCD design approaches that actively involve users is essential for enhancing usability and addressing identified UX challenges. Integrating with clinical systems is crucial for interoperability and presents opportunities for AI-enabled CDSS that rely on large patient data. Incorporating emerging technologies such as Explainable Artificial Intelligence can boost trust and acceptance. Enabling functionality for CDSS to support both clinicians and patients can create opportunities for effective use in virtual care.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The PRISMA 2020 statement: An updated guideline for reporting systematic reviews
Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n71 · Cited by 108,283
The Discovery of Grounded Theory: Strategies for Qualitative Research
Glaser, B. G., & Strauss, A. L. (2017) · Aldine (Routledge reprint) · doi:10.4324/9780203793206 · Cited by 25,789
Co-creation and the new landscapes of design
Sanders, E. B.-N., & Stappers, P. J. (2008) · CoDesign · doi:10.1080/15710880701875068 · Cited by 5,561 · cited 1×
“To achieve this, adoption of user-centered design (UCD) methodologies [15, 16], which emphasize active user engagement to understand and address their unique needs throughout the design, testing, and integration phases, offers significant advantages.”
Lost in knowledge translation: Time for a map?
Graham, I. D., Logan, J., Harrison, M. B., Straus, S. E., Tetroe, J., Caswell, W., & Robinson, N. (2006) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.47 · Cited by 4,705 · cited 1×
“In contrast, the knowledge-to-action framework focuses on knowledge creation and translation, engaging end users at specific framework stages [35].”
An overview of clinical decision support systems: Benefits, risks, and strategies for success
Sutton, R. T., Pincock, D., Baumgart, D. C., Sadowski, D. C., Fedorak, R. N., & Kroeker, K. I. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0221-y · Cited by 3,019 · cited 5×
“For instance, integrating CDSS with personal health record (PHR) systems would allow direct retrieval of patient records, while AI-enabled CDSS could access data from these integrated systems for training, predictive modeling, and decision support [2,95].”
InfluentialGarg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936
Greenhalgh, T., Wherton, J., Papoutsi, C., Lynch, J., Hughes, G., A'Court, C., Hinder, S., Fahy, N., Procter, R., & Shaw, S. (2017) · Journal of Medical Internet Research · doi:10.2196/jmir.8775 · Cited by 2,887 · cited 1×
“The way a technology is designed impacts user engagement, interaction modalities, and its ability to meet specific user needs [11,12].”
Effect of Clinical Decision-Support Systems: A Systematic Review
Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-1-201207030-00450 · Cited by 1,200 · cited 1×
“Studies show that CDSS improve clinician performance [3-5], assist with medical condition management [2,6], and enhance screening procedures [7, 8].”
Using grounded theory as a method for rigorously reviewing literature
Wolfswinkel, J. F., Furtmueller, E., & Wilderom, C. P. M. (2013) · European Journal of Information Systems · doi:10.1057/ejis.2011.51 · Cited by 1,154 · cited 2×
“For the analysis data, we followed the GTLR approach, which consists of 3 key analytical coding steps: open coding, axial coding, and selective coding [30].”
The ethics of AI in health care: A mapping review
Morley, J., Machado, C. C. V., Burr, C., Cowls, J., Joshi, I., Taddeo, M., & Floridi, L. (2020) · Social Science & Medicine · doi:10.1016/j.socscimed.2020.113172 · Cited by 1,105 · cited 1×
“…challenges in accessing requisite and curated data, particularly for AI-enabled CDSS. Privacy concerns necessitate stringent legal protocols for patient data acquisition, while compatibility issues related to data formatting and extraction further complicate access for AI-driven CDSS [92-94].”
SMART on FHIR: a standards-based, interoperable apps platform for electronic health records
Mandel, J. C., Kreda, D. A., Mandl, K. D., Kohane, I. S., & Ramoni, R. B. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv189 · Cited by 829 · cited 2×
“Moreover, these interoperability standards formalize clinical data capture standards and formats, thereby alleviating challenges related to data formatting and curation [82,96].”
Key principles for user-centred systems design
Gulliksen, J., Göransson, B., Boivie, I., Blomkvist, S., Persson, J., & Cajander, Å. (2003) · Behaviour & Information Technology · doi:10.1080/01449290310001624329 · Cited by 749 · cited 2×
“Furthermore, the iterative usability testing and evaluation stages within UCD provide substantial opportunities for user testing and refining technical features, such as decision algorithms, to ensure reliability, suitability, functionality, and appropriateness [14,15,34].”
Clinical decision support systems for the practice of evidence-based medicine
Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., & Tang, P. C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080527 · Cited by 732 · cited 1×
“Clinical decision support systems (CDSS) encompass both computerized and noncomputerized tools designed to improve and support health care delivery [1].”
Johnston, M. E., Langton, K. B., Haynes, R. B., & Mathieu, A. (1994) · Annals of Internal Medicine · doi:10.7326/0003-4819-120-2-199401150-00007 · Cited by 714 · cited 1×
“Research has highlighted their benefits in improving primary health care and enhancing physician performance [18-20].”
Vasey, B., Nagendran, M., Campbell, B., Clifton, D. A., Collins, G. S., Denaxas, S., Denniston, A. K., Faes, L., Geerts, B., Ibrahim, M., et al. (2022) · BMJ · doi:10.1136/bmj-2022-070904 · Cited by 665 · cited 1×
“In relation to the usability evaluation of CDSS, studies have revealed inconsistencies due to a lack of standardized assessment processes [24], leading researchers to propose diverse evaluation methods [25,26].”
Antoniadi, A. M., Du, Y., Guendouz, Y., Wei, L., Mazo, C., Becker, B. A., & Mooney, C. (2021) · Applied Sciences · doi:10.3390/app11115088 · Cited by 629 · cited 1×
“The utilization of emerging technologies such as XAI [39,43,93,105] holds promise in addressing these challenges by ensuring human comprehension of AI decisions.”
HL7 FHIR: An Agile and RESTful approach to healthcare information exchange
Bender, D., & Sartipi, K. (2013) · Proceedings of the 26th IEEE International Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2013.6627810 · Cited by 540 · cited 1×
“Moreover, these interoperability standards formalize clinical data capture standards and formats, thereby alleviating challenges related to data formatting and curation [82,96].”
Jaspers, M. W. M., Smeulers, M., Vermeulen, H., & Peute, L. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000094 · Cited by 525 · cited 1×
“Studies show that CDSS improve clinician performance [3-5], assist with medical condition management [2,6], and enhance screening procedures [7, 8].”
User-Centered Design and Interactive Health Technologies for Patients
DABBS, A. D. V., MYERS, B. A., MC CURRY, K. R., DUNBAR-JACOB, J., HAWKINS, R. P., BEGEY, A., & DEW, M. A. (2009) · CIN: Computers, Informatics, Nursing · doi:10.1097/ncn.0b013e31819f7c7c · Cited by 517 · cited 1×
“Previous studies [11,14, 17,88,89] corroborate the effectiveness of UCD in eliciting user needs, workflow requirements, and efficiency in CDSS design.”
Kwan, J. L., Lo, L., Ferguson, J., Goldberg, H., Diaz-Martinez, J. P., Tomlinson, G., Grimshaw, J. M., & Shojania, K. G. (2020) · BMJ · doi:10.1136/bmj.m3216 · Cited by 461 · cited 1×
“Their widespread adoption has contributed to improved patient-clinician interactions, reduced medical errors, and increased adherence to clinical guidelines [9,10].”
Machine Learning and Decision Support in Critical Care
Johnson, A. E. W., Ghassemi, M. M., Nemati, S., Niehaus, K. E., Clifton, D., & Clifford, G. D. (2016) · Proceedings of the IEEE · doi:10.1109/jproc.2015.2501978 · Cited by 432 · cited 1×
“Machine learning–based systems develop knowledge from existing clinical data to apply it to new datasets [78, 79].”
Security and Privacy-Preserving Challenges of e-Health Solutions in Cloud Computing
Chenthara, S., Ahmed, K., Wang, H., & Whittaker, F. (2019) · IEEE Access · doi:10.1109/access.2019.2919982 · Cited by 335 · cited 1×
“While vast clinical datasets may be available, navigating access protocols can be cumbersome due to data access privacy concerns [85,86].”
Big data security and privacy in healthcare: A Review
Abouelmehdi, K., Beni-Hssane, A., Khaloufi, H., & Saadi, M. (2017) · Procedia Computer Science · doi:10.1016/j.procs.2017.08.292 · Cited by 264 · cited 1×
“While vast clinical datasets may be available, navigating access protocols can be cumbersome due to data access privacy concerns [85,86].”
Amann, J., Vetter, D., Blomberg, S. N., Christensen, H. C., Coffee, M., Gerke, S., Gilbert, T. K., Hagendorff, T., Holm, S., Livne, M., et al. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000016 · Cited by 251 · cited 1×
“…systems offer promising solutions to address trust and validation challenges in CDSS. XAI enhances transparency by making AI systems’ inner workings more interpretable for health care professionals, enabling them to identify potential flaws and assess the trustworthiness of system outputs [83].”
Harnessing the Power of Clinical Decision Support Systems: Challenges and Opportunities
Chen, Z., Liang, N., Zhang, H., Li, H., Yang, Y., Zong, X., Chen, Y., Wang, Y., & Shi, N. (2023) · Open Heart · doi:10.1136/openhrt-2023-002432 · Cited by 244 · cited 1×
“Additionally, studies have examined the factors affecting CDSS adoption, identifying challenges such as privacy concerns [21], integration issues [22], and quality-in-use limitations [23].”
Prgomet, M., Li, L., Niazkhani, Z., Georgiou, A., & Westbrook, J. I. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw145 · Cited by 243
Wang, D., Wang, L., Zhang, Z., Wang, D., Zhu, H., Gao, Y., Fan, X., & Tian, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445432 · Cited by 209 · cited 1×
“…challenges in accessing requisite and curated data, particularly for AI-enabled CDSS. Privacy concerns necessitate stringent legal protocols for patient data acquisition, while compatibility issues related to data formatting and extraction further complicate access for AI-driven CDSS [92-94].”
Clinical decision support systems
Beeler, P. E., Bates, D. W., & Hug, B. L. (2014) · Swiss Medical Weekly · doi:10.4414/smw.2014.14073 · Cited by 188 · cited 1×
“For instance, integrating CDSS with personal health record (PHR) systems would allow direct retrieval of patient records, while AI-enabled CDSS could access data from these integrated systems for training, predictive modeling, and decision support [2,95].”
Taheri Moghadam, S., Sadoughi, F., Velayati, F., Ehsanzadeh, S. J., & Poursharif, S. (2021) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-020-01376-8 · Cited by 136
Co-design of human-centered, explainable AI for clinical decision support
Panigutti, C., Beretta, A., Fadda, D., Giannotti, F., Pedreschi, D., Perotti, A., & Rinzivillo, S. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3587271 · Cited by 127 · cited 8×
“The utilization of emerging technologies such as XAI [39,43,93,105] holds promise in addressing these challenges by ensuring human comprehension of AI decisions.”
InfluentialUser-centered design to improve clinical decision support in primary care
Brunner, J., Chuang, E., Goldzweig, C., Cain, C. L., Sugar, C., & Yano, E. M. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.05.004 · Cited by 117 · cited 2×
“Previous studies [11,14, 17,88,89] corroborate the effectiveness of UCD in eliciting user needs, workflow requirements, and efficiency in CDSS design.”
Segal, G., Segev, A., Brom, A., Lifshitz, Y., Wasserstrum, Y., & Zimlichman, E. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz135 · Cited by 116 · cited 5×
“Key fundamental challenges related to their validation, including determining where to initiate the validation process, establishing a “ground truth” for validation, and defining a criteria for success in validation [54,60,64,71].”
InfluentialUser-centered methods for designing patient-centric self-help tools
Årsand, E., & Demiris, G. (2008) · Informatics for Health and Social Care · doi:10.1080/17538150802457562 · Cited by 114 · cited 1×
“Effective user engagement and participation also foster a sense of empowerment and ownership [90,91], enhancing user understanding of the system’s operation and functionality, thereby addressing the trust and adoption-related concerns identified.”
Miller, A., Moon, B., Anders, S., Walden, R., Brown, S., & Montella, D. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.09.005 · Cited by 106 · cited 1×
“Additionally, studies have examined the factors affecting CDSS adoption, identifying challenges such as privacy concerns [21], integration issues [22], and quality-in-use limitations [23].”
The Impact of Clinical Decision Support Systems (CDSS) on Physicians: A Scoping Review
Muhiyaddin, R., Abd-Alrazaq, A. A., Househ, M. S., Alam, T., & Shah, Z. (2020) · Studies in health technology and informatics · doi:10.3233/shti200597 · Cited by 95 · cited 1×
“Research has highlighted their benefits in improving primary health care and enhancing physician performance [18-20].”
A Human(e) Factor in Clinical Decision Support Systems
Bezemer, T., de Groot, M., Blasse, E., ten Berg, M. J., Kappen, T. H., Bredenoord, A. L., van Solinge, W. W., Hoefer, I. E., & Haitjema, S. (2019) · Journal of Medical Internet Research · doi:10.2196/11732 · Cited by 93
Keikhosrokiani, P., Mustaffa, N., Zakaria, N., & Abdullah, R. (2019) · Cognition Technology & Work · doi:10.1007/s10111-019-00565-4 · Cited by 88 · cited 1×
“These factors, in turn, shape user experience (UX), adoption rates, and perceived usefulness [13,14].”
A distributed clinical decision support system architecture
El–Sappagh, S., & El‐Masri, S. (2013) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2013.03.005 · Cited by 75 · cited 1×
“This approach enhances the overall efficacy and performance of the system, particularly in the context of integration [63,97, 98].”
Luna, D. R., Lede, D. A. R., Otero, C., Risk, M. R., & de Quirós, F. G. B. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.01.009 · Cited by 75 · cited 8×
“Key fundamental challenges related to their validation, including determining where to initiate the validation process, establishing a “ground truth” for validation, and defining a criteria for success in validation [54,60,64,71].”
InfluentialYin, Z., Dong, Z., Lü, X., Yu, S., Chen, X., & Duan, H. (2015) · The Journal of Headache and Pain · doi:10.1186/s10194-015-0512-x · Cited by 70 · cited 2×
“All CDSS examined in the reviewed papers underwent some form of usability evaluation or testing to assess functionality [42,51-54], feasibility [38,40,49], acceptability [37,38,47,49,51,52,55-58], accuracy [41,58-61] , quality of experience [50,62-65], and usefulness [60,65,66].”
Tanguay-Sela, M., Benrimoh, D. A., Popescu, C., Pérez, T., Rollins, C. P. E., Snook, E., Lundrigan, E., Armstrong, C., Perlman, K., Fratila, R., et al. (2021) · Psychiatry Research · doi:10.1016/j.psychres.2021.114336 · Cited by 69
Moja, L., Polo Friz, H., Capobussi, M., Kwag, K., Banzi, R., Ruggiero, F., González-Lorenzo, M., Liberati, E. G., Mangia, M., Nyberg, P., et al. (2019) · JAMA Network Open · doi:10.1001/jamanetworkopen.2019.17094 · Cited by 68
Towards user empowerment in product design: a mixed reality tool for interactive virtual prototyping
Arrighi, P.-A., & Mougenot, C. (2016) · Journal of Intelligent Manufacturing · doi:10.1007/s10845-016-1276-0 · Cited by 65 · cited 1×
“Effective user engagement and participation also foster a sense of empowerment and ownership [90,91], enhancing user understanding of the system’s operation and functionality, thereby addressing the trust and adoption-related concerns identified.”
Bider, I., & Jalali, A. (2014) · Information Systems and e-Business Management · doi:10.1007/s10257-014-0256-1 · Cited by 62 · cited 1×
“The agile business process development approach takes an integrated business process model to guide technology design and implementation [36].”
A Survey on Case-based Reasoning in Medicine
Choudhury, N., & Ara, S. (2016) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2016.070820 · Cited by 62 · cited 1×
“Case-based reasoning systems compare new cases with stored past decisions to identify similarities and determine outcomes [80].”
Wronikowska, M. W., Malycha, J., Morgan, L. J., Westgate, V., Petrinic, T., Young, J. D., & Watkinson, P. J. (2021) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.13582 · Cited by 59 · cited 1×
“In relation to the usability evaluation of CDSS, studies have revealed inconsistencies due to a lack of standardized assessment processes [24], leading researchers to propose diverse evaluation methods [25,26].”
Patient perceptions of the benefits and barriers of virtual postnatal care: a qualitative study
Saad, M., Chan-Nguyen, S., Nguyen, L., Srivastava, S., & Appireddy, R. (2021) · BMC Pregnancy and Childbirth · doi:10.1186/s12884-021-03999-9 · Cited by 58 · cited 1×
“…the implementation of CDSS that facilitate the remote provision of patient vital signs (a requirement of most CDSS) could obviate the need for physical visits to medical facilities, thereby providing comfort and reducing health care delivery costs for both patients and health care providers [103].”
Wright, A., Sittig, D. F., Ash, J. S., Erickson, J. L., Hickman, T. T., Paterno, M., Gebhardt, E., McMullen, C., Tsurikova, R., Dixon, B. E., et al. (2015) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2015.08.008 · Cited by 55 · cited 5×
“Our findings indicate that most CDSS were stand-alone systems, with only 20% (8/40) of the CDSS reviewed being integrated into EHRs and existing workflows, and 17.5% (7/40) of them being intentionally designed with integration as a core feature [37,42,44,45,49,52-55, 60,”
InfluentialDevelopment of a clinical decision support system for diabetes care: A pilot study
Sim, L. L. W., Ban, K. H.-K., Tan, T. W., Sethi, S. K., & Loh, T. P. (2017) · PLoS ONE · doi:10.1371/journal.pone.0173021 · Cited by 55 · cited 7×
“Moreover, modelling the clinical workflow for integration into the design of CDSS proves difficult due to the dynamic and constantly evolving nature of the clinical environment [51,71].”
InfluentialDesigning for User Experience and Engagement
Sutcliffe, A. (2016) · Springer eBooks · doi:10.1007/978-3-319-27446-1_5 · Cited by 52 · cited 2×
“Previous studies [11,14, 17,88,89] corroborate the effectiveness of UCD in eliciting user needs, workflow requirements, and efficiency in CDSS design.”
Romero‐Aroca, P., Valls, A., Moreno, A., Sagarra-Alamo, R., Basora, J., Saleh, E., Baget‐Bernaldiz, M., & Puig, D. (2018) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2017.0282 · Cited by 50
ConfidentCare: A Clinical Decision Support System for Personalized Breast Cancer Screening
Alaa, A. M., Moon, K. H., Hsu, W., & van der Schaar, M. (2016) · IEEE Transactions on Multimedia · doi:10.1109/tmm.2016.2589160 · Cited by 47 · cited 1×
“Studies show that CDSS improve clinician performance [3-5], assist with medical condition management [2,6], and enhance screening procedures [7, 8].”
Choi, A., Choi, S. Y., Chung, K., Chung, H. S., Song, T., Choi, B., & Kim, J. H. (2023) · Scientific Reports · doi:10.1038/s41598-023-35617-3 · Cited by 47 · cited 1×
“Machine learning–based systems develop knowledge from existing clinical data to apply it to new datasets [78, 79].”
Virtual hospitals: The future of the healthcare system? An expert consensus
Bidoli, C., Pegoraro, V., Mas, F. D., Bagnoli, C., Bert, F., Bonin, M., Butturini, G., Cobianchi, L., Cordiano, C., Minto, G., et al. (2023) · Journal of Telemedicine and Telecare · doi:10.1177/1357633x231173006 · Cited by 40 · cited 1×
“While this emphasis reinforces the critical role of CDSS in supporting clinical decision-making, it also limits their broader applicability in areas such as virtual care and integration with PHRs [99-101].”
Ray, J. M., Ahmed, O. M., Solad, Y., Maleska, M., Martel, S. H., Jeffery, M. M., Platts‐Mills, T. F., Hess, E. P., D’Onofrio, G., & Melnick, E. R. (2019) · JMIR Human Factors · doi:10.2196/13121 · Cited by 35 · cited 2×
“Several studies highlighted limitations in the presentation design of CDSS recommendations, noting rigid structures that restrict users from incorporating their own perspectives [45,46,70, 74].”
Human Computer Interaction and medical devices
Acharya, C., Thimbleby, H., & Oladimeji, P. (2010) · Electronic workshops in computing · doi:10.14236/ewic/hci2010.22 · Cited by 32 · cited 3×
“These factors, in turn, shape user experience (UX), adoption rates, and perceived usefulness [13,14].”
Breitbart, E., Choudhury, K., Andersen, A. D., Bunde, H., Breitbart, M., Sideri, A. M., Fengler, S., & Zibert, J. (2020) · PLoS ONE · doi:10.1371/journal.pone.0235410 · Cited by 31 · cited 2×
“…vital signs data such as temperature, blood pressure, and heart rate [38,46,47,54,58,62, 63,65,66]; laboratory data (test results) [41,51,53,56,59,66,73, 76]; EHRs (specific or combinations of information such as clinical history, comorbidities, or prescriptions) [37,42, 44,45,49,52-55,60…”
Farmer, N. (2014) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.12153 · Cited by 30 · cited 3×
“…relay of output feedback, ensure visibility, engagement, and adherence; various alert and notification mechanisms, including prompts such as beeps or pop-ups [46,53,55,67], color coding of recommendations to signify severity [40,41, 51], and sending messages as reminders [55,64], are used.”
Bau, C.-T., Chen, R.-C., & Huang, C.-Y. (2014) · Telemedicine Journal and e-Health · doi:10.1089/tmj.2013.0221 · Cited by 30 · cited 2×
“However, there was a notable absence of ontology-based reasoning systems among the reviewed papers, likely due to the emerging nature of this category of CDSS [81].”
Wohlgemut, J. M., Pisirir, E., Kyrimi, E., Stoner, R. S., Marsh, W., Perkins, Z., & Tai, N. R. M. (2023) · JAMIA Open · doi:10.1093/jamiaopen/ooad051 · Cited by 30 · cited 1×
“In relation to the usability evaluation of CDSS, studies have revealed inconsistencies due to a lack of standardized assessment processes [24], leading researchers to propose diverse evaluation methods [25,26].”
Zon, M., Ganesh, G. M., Deen, M. J., & Fang, Q. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20146399 · Cited by 29 · cited 1×
“Additionally, context-aware machine learning systems that integrate patient-specific and environmental factors can generate more personalized and relevant recommendations, thereby reducing ambiguity and fostering trust in CDSS [84].”
Gibbons, M. C., Lowry, S. Z., & Patterson, E. S. (2014) · JMIR Human Factors · doi:10.2196/humanfactors.3524 · Cited by 28 · cited 1×
“Furthermore, reviews focusing on human factors have identified challenges and barriers such as user fatigue, which can significantly impact usability and long-term adoption [27,28].”
Quality-in-use characteristics for clinical decision support system assessment
Souza-Pereira, L., Ouhbi, S., & Pombo, N. G. C. C. (2021) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2021.106169 · Cited by 28 · cited 1×
“Additionally, studies have examined the factors affecting CDSS adoption, identifying challenges such as privacy concerns [21], integration issues [22], and quality-in-use limitations [23].”
Virtual Care 2.0—a Vision for the Future of Data-Driven Technology-Enabled Healthcare
Bhavnani, S. P., & Sitapati, A. M. (2019) · Current Treatment Options in Cardiovascular Medicine · doi:10.1007/s11936-019-0727-2 · Cited by 27
Nair, K., Malaeekeh, R., Schabort, I., Taenzer, P. A., Radhakrishnan, A., & Guenter, D. (2015) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v22i3.149 · Cited by 26
Conway, N., Adamson, K., Cunningham, S. G., Smith, A. E., Nyberg, P. M., Smith, B. H., Wales, A., & Wake, D. J. (2017) · Journal of Diabetes Science and Technology · doi:10.1177/1932296817729489 · Cited by 26 · cited 9×
“Design or integration complexities • Transforming guidelines decision rules [41,54,55].”
InfluentialVirtual care in the health care system: A concept analysis
Babaei, N., Zamanzadeh, V., Valizadeh, L., Lotfi, M., Kousha, A., Samad‐Soltani, T., & Avazeh, M. (2023) · Scandinavian Journal of Caring Sciences · doi:10.1111/scs.13227 · Cited by 26 · cited 1×
“Considering the economic, accessibility, and efficiency benefits offered by technology-mediated health care options such as virtual care [102], CDSS present opportunities for adoption in such health care delivery schemes.”
Beerlage-de Jong, N., Wentzel, J., Hendrix, R., & van Gemert-Pijnen, L. (2017) · American Journal of Infection Control · doi:10.1016/j.ajic.2016.12.001 · Cited by 24 · cited 8×
“Users expressed concerns regarding how recommendations are presented and accessed [46] and inconsistencies in workflow integration [49,62].”
InfluentialGarbern, S. C., Nelson, E. J., Nasrin, S., Keita, A. M., Brintz, B. J., Gainey, M., Badji, H., Nasrin, D., Howard, J., Taniuchi, M., et al. (2022) · eLife · doi:10.7554/elife.72294 · Cited by 21
Healthcare and the Effect of Technology: Developments, Challenges and Advancements
Kabene, S. (2010) · Advances in healthcare information systems and administration book series · doi:10.4018/978-1-61520-733-6 · Cited by 19 · cited 1×
“With the growing implementation and adoption of national health records across countries [2,104], PHR would become more accessible for leverage by patient-facing CDSS.”
Zhang, X., Svec, M., Tracy, R., & Ozanich, G. (2021) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2021.104626 · Cited by 18 · cited 4×
“All CDSS examined in the reviewed papers underwent some form of usability evaluation or testing to assess functionality [42,51-54], feasibility [38,40,49], acceptability [37,38,47,49,51,52,55-58], accuracy [41,58-61] , quality of experience [50,62-65], and usefulness [60,65,66].”
InfluentialKnight, S. R., Cao, K. N., South, M., Hayward, N., Hunter, J. P., & Fox, J. (2018) · Transplantation · doi:10.1097/tp.0000000000002374 · Cited by 17 · cited 4×
“Several studies highlighted limitations in the presentation design of CDSS recommendations, noting rigid structures that restrict users from incorporating their own perspectives [45,46,70, 74].”
InfluentialDevelopment of a Knowledge-based Clinical Decision Support System for Multiple Sclerosis Diagnosis
Hosseini, A., Asadi, F., & Arani, L. A. (2020) · Journal of Medicine and Life · doi:10.25122/jml-2020-0182 · Cited by 17 · cited 3×
“…vital signs data such as temperature, blood pressure, and heart rate [38,46,47,54,58,62, 63,65,66]; laboratory data (test results) [41,51,53,56,59,66,73, 76]; EHRs (specific or combinations of information such as clinical history, comorbidities, or prescriptions) [37,42, 44,45,49,52-55,60 As…”
Abtahi, H., Amini, S., Gholamzadeh, M., & Gharabaghi, M. A. (2023) · Informatics in Medicine Unlocked · doi:10.1016/j.imu.2023.101168 · Cited by 16 · cited 9×
“• Surveys [37,50,57,73] Usability testing or evaluation • Think Aloud protocol [41,52].”
InfluentialShahmoradi, L., Safdari, R., Mirhosseini, M. M., Rezayi, S., & Javaherzadeh, M. (2023) · Scientific Reports · doi:10.1038/s41598-023-46721-9 · Cited by 16 · cited 3×
“Also, a significant trend observed in the review was the increasing use of AI and machine learning–based approaches to develop predictive decision support systems [39,43,47,48, 53,57,60,61,73,75].”
Explainable AI-based clinical decision support system for hearing disorders.
Tarnowska, K. A., Dispoto, B. C., & Conragan, J. (2021) · PubMed · Cited by 16
A Knowledge-Based Variance Management System for Supporting the Implementation of Clinical Pathways
Ye, Y., Diao, X., Jiang, Z., & Du, G. (2009) · International Conference on Management and Service Science (MASS) · doi:10.1109/icmss.2009.5305190 · Cited by 15 · cited 1×
“Rule-based systems use deterministic “if-then-else” rules for decision-making [77].”
Shalom, E., Goldstein, A., Ariel, E., Sheinberger, M., Jones, V. M., van Schooten, B. W., & Shahar, Y. (2022) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2022.102324 · Cited by 15 · cited 2×
“…encompassed combinations of 5 categories of data sources: vital signs data such as temperature, blood pressure, and heart rate [38,46,47,54,58,62, 63,65,66]; laboratory data (test results) [41,51,53,56,59,66,73, 76]; EHRs (specific or combinations of information such as clinical history,…”
How the electronic health record will change the future of health care.
Ronquillo, J. G. (2012) · Europe PMC (PubMed Central) · Cited by 14
Kropf, M., Modre‐Osprian, R., Hayn, D., Fruhwald, F. M., & Schreier, G. (2014) · Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) · doi:10.1109/embc.2014.6944295 · Cited by 13 · cited 10×
“Translating guidelines into decision rules that are implementable within CDSS presents nontrivial challenges [41,54, 55], often requiring extensive data collection over extended periods.”
InfluentialKHONG, P. C. B., Lee, L. W., & Dawang, A. (2017) · Informatics · doi:10.3390/informatics4030020 · Cited by 13 · cited 4×
“…[41,51,53,56,59,66,73, 76]; EHRs (specific or combinations of information such as clinical history, comorbidities, or prescriptions) [37,42, 44,45,49,52-55,60 As output, systems deliver recommendations on treat-ment plans, workflow, or dosage [49,54,64,74]; reports on conditions including…”
InfluentialPaydar, P., Ebrahimpour, S., Hashemi, H. Z., Mohamadi, M., & Namazi, S. (2023) · PubMed Central · doi:10.30476/jamp.2022.95843.1661 · Cited by 13 · cited 2×
“Also, a significant trend observed in the review was the increasing use of AI and machine learning–based approaches to develop predictive decision support systems [39,43,47,48, 53,57,60,61,73,75].”
Alhodaib, H. I., Antza, C., Chandan, J. S., Hanif, W., Sankaranarayanan, S., Paul, S., Sutcliffe, P. A., & Nirantharakumar, K. (2020) · JMIR Diabetes · doi:10.2196/19650 · Cited by 12 · cited 6×
“• Questionnaires [41,46,55,65,68].”
InfluentialTowards an explainable clinical decision support system for large-for-gestational-age births
Du, Y., Rafferty, A. R., McAuliffe, F. M., Mehegan, J., & Mooney, C. (2023) · PLoS ONE · doi:10.1371/journal.pone.0281821 · Cited by 11 · cited 1×
“The utilization of emerging technologies such as XAI [39,43,93,105] holds promise in addressing these challenges by ensuring human comprehension of AI decisions.”
Siek, K. A. (2018) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1667073 · Cited by 10 · cited 1×
“Previous studies [11,14, 17,88,89] corroborate the effectiveness of UCD in eliciting user needs, workflow requirements, and efficiency in CDSS design.”
Decision Support in Evaluating the Impacts of Mental Disorders on Work Ability
Kemppinen, J., Korpela, J., Elfvengren, K., Salmisaari, T., & Polkko, J. (2014) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2014.368 · Cited by 9 · cited 5×
“Furthermore, a mismatch between users’ practical workflow and the requirements of CDSS poses significant challenges [42,49,51,62,76].”
InfluentialClinical Decision Support System in Virtual Clinic
Rahman, A., Bakry, A., Sultan, K., Khan, M. A. A., Farooqui, M., & Musleh, D. A. (2018) · Journal of Computational and Theoretical Nanoscience · doi:10.1166/jctn.2018.7313 · Cited by 9 · cited 1×
“While this emphasis reinforces the critical role of CDSS in supporting clinical decision-making, it also limits their broader applicability in areas such as virtual care and integration with PHRs [99-101].”
Using Task Technology Fit Theory to Guide the Codesign of Mobile Clinical Decision Support Systems
Ulapane, N. N. B., Forkan, A. R. M., Jayaraman, P. P., Schofield, P. E., Burbury, K., & Wickramasinghe, N. S. (2023) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2023.353 · Cited by 9 · cited 7×
“• Surveys [37,50,57,73] Usability testing or evaluation • Think Aloud protocol [41,52].”
InfluentialZhao, Y., Hu, J., Gu, Y., Wan, Y., Liu, F., Ye, C., & Xiaobo, Z. (2021) · CIN Computers Informatics Nursing · doi:10.1097/cin.0000000000000812 · Cited by 8
Development of a Clinical Decision Support System for Smart Algorithms in Emergency Medicine
Vogel, S., Reiswich, A., Ritter, Z., Schmucker, M., Fuchs, A., Pischek-Koch, K., Wache, S., Esslinger, K., Dietrich, M., Kesztyüs, T., et al. (2022) · Studies in health technology and informatics · doi:10.3233/shti210900 · Cited by 8 · cited 1×
“Also, a significant trend observed in the review was the increasing use of AI and machine learning–based approaches to develop predictive decision support systems [39,43,47,48, 53,57,60,61,73,75].”
Steffny, L., Dahlem, N., Reichl, L., Gisa, K., Greff, T., & Werth, D. (2023) · Frontiers in Artificial Intelligence and Applications · doi:10.3233/faia230090 · Cited by 8 · cited 5×
“The utilization of emerging technologies such as XAI [39,43,93,105] holds promise in addressing these challenges by ensuring human comprehension of AI decisions.”
InfluentialBarigou, B. N., Barigou, F., Benchehida, C., Atmani, B., & Belalem, G. (2018) · International Journal of Healthcare Information Systems and Informatics · doi:10.4018/ijhisi.2018100103 · Cited by 7
Virtual data integration for a clinical decision support systems
Nasir, I. S., Mousa, A. H., Alkhafaji, S. M. A., Hussein, W. S. A., Jasim, Z. R., & Ali, S. Q. (2023) · International Journal of Power Electronics and Drive Systems/International Journal of Electrical and Computer Engineering · doi:10.11591/ijece.v13i5.pp5243-5252 · Cited by 7
Seitinger, A., Fehre, K., Adlassnig, K.-P., Rappelsberger, A., Wurm, E. M. T., Aberer, E., & Binder, M. (2014) · PubMed · Cited by 7
Distributed gene clinical decision support system based on cloud computing
Xu, B., Li, C., Zhuang, H., Wang, J., Wang, Q., Wang, C., & Zhou, X. (2018) · BMC Medical Genomics · doi:10.1186/s12920-018-0415-1 · Cited by 6 · cited 1×
“This approach enhances the overall efficacy and performance of the system, particularly in the context of integration [63,97, 98].”
Malik, T. F. A., Teo, C. H., Teh, S. Y., Su, J. Y., Zaini, H. S. M., Suhaimi, A., Ng, W. L., Danaee, M., Stevenson, K., Mallen, C. D., et al. (2023) · Family Practice · doi:10.1093/fampra/cmad044 · Cited by 6 · cited 9×
“Users expressed concerns regarding how recommendations are presented and accessed [46] and inconsistencies in workflow integration [49,62].”
InfluentialFerreira, G., Oliveira, E., Stamper, J. C., Coelho, A. F., Paredes, H., & Rodrigues, N. F. (2023) · SeGAH ... IEEE ... International Conference on Serious Games and Applications for Health/IEEE International Conference on Serious Games and Applications for Health · doi:10.1109/segah57547.2023.10253790 · Cited by 5 · cited 1×
“Furthermore, reviews focusing on human factors have identified challenges and barriers such as user fatigue, which can significantly impact usability and long-term adoption [27,28].”
Towards a Clinical Decision Support System for Helping Medical Students in Emergency Call Centers
Michot, E., Woo, J., Mouline, L., Sinnappan, C., Boukobza, A., Campeotto, F., Dupic, L., Burgun, A., Vivien, B., & Tsopra, R. (2022) · Studies in health technology and informatics · doi:10.3233/shti220494 · Cited by 3 · cited 2×
“Furthermore, trust issues arise, as more experienced clinicians may disregard CDSS recommendations when they conflict with their own clinical judgment or opinions [45,72,74].”
Clinical Decision Support System for Diagnosis and Treatment of COPD Using Ensemble Methods
Anakal, S. S., & Sandhya, P. (2023) · Studies in computational intelligence · doi:10.1007/978-981-99-1482-1_1 · Cited by 3
Usability Enhancements to a Prototype Clinical Decision Support System for Combat Medics
Gaudaen, J. C., Papadopoulos, A., Dockery, T., & Manemeit, C. (2023) · Military Medicine · doi:10.1093/milmed/usad279 · Cited by 3 · cited 6×
“• Personas [40].”
InfluentialKieffer, S., Gouze, A., & Vanderdonckt, J. M. (2022) · SN Computer Science · doi:10.1007/s42979-022-01451-z · Cited by 2 · cited 3×
“• Contextual and integration challenges [37,39,66,68,71,74].”
Evaluating the QoE of a mobile DSS for diagnosis of red eye diseases by medical students
López, M. M., López, M. M., Díez, I. D. L. T., Jimeno, J. C. P., López-Coronado, M., & Rodrigues, J. J. P. C. (2016) · IEEE International Conference on e-Health Networking, Applications and Services (Healthcom) · doi:10.1109/healthcom.2016.7749498 · Cited by 1 · cited 4×
“These techniques supported different aspects of the design process: informing the overall design [38,40-43], exploring user needs and workflow [44-46], contextualizing and implementing design [47-49], and ensuring usability, functionality, and effectiveness [50-52].”
Influential
Research works printed but not linked: 5
- Webster, MIS Q, 26, xiii
- Tarnowska, AMIA Jt Summits Transl Sci Proc, 2021
- Seitinger, Stud Health Technol Inform, 198
- Silva EAT Gomez IFL Arango JFF Smith JW Ocampo SU Hidalgo JE . Evaluation of satisfaction and usability of a clinical decision support system (CDSS) targeted for early obstetric risk assessment and patient follow-up. Presented at: Proceedings of the IADIS International Conference e-Health 2018; Madrid. 2018.URL: https://bit.ly/44PHend [Accessed 16-06-2025]
- Ronquillo, Yale J Biol Med, 85, 379
Cited by
Abbas, Q., Jeong, W., & Lee, S. W. (2025) · Healthcare · doi:10.3390/healthcare13172154 · Cited by 236
“Scalability, data interoperability, and usability are critical factors that determine whether these systems can transition from research prototypes to clinical tools [36,37].”
Yu, H., Chen, T., Zhang, X., Yang, Y., Liu, Q., Yang, C., Shen, K., Li, H., Tang, W., Zhong, X., et al. (2025) · Journal of Medical Internet Research · doi:10.2196/77334 · Cited by 21
“They have been extensively applied in areas such as medication review, sepsis alerts, and risk prediction for deep vein thrombosis and pressure ulcers, including those designed to assist diagnosis [46,47].”
Thorakkattil, S. A., Sridhar, S. B., Abdulsalim, S., Karattuthodi, M. S., Chandra, P., & Unnikrishnan, M. K. (2025) · Healthcare · doi:10.3390/healthcare13172220 · Cited by 10
“Additionally, we reviewed the official websites of Indian hospitals to identify announcements regarding CDSS deployments [12–17].”
An interpretable deep learning framework for medical diagnosis using spectrogram analysis
Henna, S., Alcaraz, J. M. L., Rathnayake, U., & Amjath, M. (2025) · Healthcare Analytics · doi:10.1016/j.health.2025.100408 · Cited by 10
Artificial intelligence in respiratory medicine: From diagnosis to treatment and future directions
Chen, Z., Tang, Z., Ewing, R. M., Belkhatir, Z., & Wang, Y. (2026) · Chinese Medical Journal - Pulmonary and Critical Care Medicine · doi:10.1016/j.pccm.2026.05.005 · Cited by 3
“56 As conceptualized in the updated operational workflow ( Fig.”
Metadata
- Authors
- Bayor, A. A., Li, J., Yang, I. A., & Varnfield, M.
- Affiliations
- Commonwealth Scientific and Industrial Research Organisation (AU)
- The University of Queensland (AU)
- Corresponding author
- Bayor, Andrew A
- Venue
- Journal of Medical Internet Research
- Publisher
- JMIR Publications Inc.
- Published
- 2025-06-20
- Volume
- 27
- Pages
- e63733-e63733
- ISSN
- 1438-8871 (ISSN-L)
- PMC
- PMC12463342
- OpenAlex
- W4409784533
- Semantic Scholar
- 79b3bb750c3df8e751b2ce87a9dd9865bf99d558
- Cited by
- 46
- References
- 99
- Citations by year
- 2026: 39 · 2025: 7
- FWCI
- 5.93
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- Locations
- Journal of Medical Internet Research (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- PubMed Central (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Keywords
- clinical decision support systems, user-centered design, user experience, usability, explainable AI, clinical workflow, reliability and validity, data quality, clinician trust, quality of care, evidence-based decision making, point-of-care decision support, system architecture, design challenges, health information technology evaluation, health information technology implementation
- MeSH terms
- User-Centered Design, Humans, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

