The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
A systematic review of trials evaluating success factors of interventions with computerised clinical decision support
Open accessAbstract
BACKGROUND: Computerised clinical decision support (CDS) can potentially better inform decisions, and it can help with the management of information overload. It is perceived to be a key component of a learning health care system. Despite its increasing implementation worldwide, it remains uncertain why the effect of CDS varies and which factors make CDS more effective. OBJECTIVE: To examine which factors make CDS strategies more effective on a number of outcomes, including adherence to recommended practice, patient outcome measures, economic measures, provider or patient satisfaction, and medical decision quality. METHODS: We identified randomised controlled trials, non-randomised trials, and controlled before-and-after studies that directly compared CDS implementation with a given factor to CDS without that factor by searching CENTRAL, MEDLINE, EMBASE, and CINAHL and checking reference lists of relevant studies. We considered CDS with any objective for any condition in any healthcare setting. We included CDS interventions that were either displayed on screen or provided on paper and that were directed at healthcare professionals or targeted at both professionals and patients. The reviewers screened the potentially relevant studies in duplicate. They extracted data and assessed risk of bias in independent pairs or individually followed by a double check by another reviewer. We summarised results using medians and interquartile ranges and rated our certainty in the evidence using the GRADE system. RESULTS: We identified 66 head-to-head trials that we synthesised across 14 comparisons of CDS intervention factors. Providing CDS automatically versus on demand led to large improvements in adherence. Displaying CDS on-screen versus on paper led to moderate improvements and making CDS more versus less patient-specific improved adherence modestly. When CDS interventions were combined with professional-oriented strategies, combined with patient-oriented strategies, or combined with staff-oriented strategies, then adherence improved slightly. Providing CDS to patients slightly increased adherence versus CDS aimed at the healthcare provider only. Making CDS advice more explicit and requiring users to respond to the advice made little or no difference. The CDS intervention factors made little or no difference to patient outcomes. The results for economic outcomes and satisfaction outcomes were sparse. CONCLUSION: Multiple factors may affect the success of CDS interventions. CDS may be more effective when the advice is provided automatically and displayed on-screen and when the suggestions are more patient-specific. CDS interventions combined with other strategies probably also improves adherence. Providing CDS directly to patients may also positively affect adherence. The certainty of the evidence was low to moderate for all factors. TRIAL REGISTRATION: PROSPERO, CRD42016033738.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Although previous studies have studied factors affecting the implementation of CDS tools [48], the use of black box models in health care settings presents unique challenges and opportunities related to trust and transparency.”
From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [48]
In the citing paper
“The WPA in common with WHO advocates for a pragmatic approach to implementing viable alternatives to coercion, 601 one of which is adopting a community-based collaborative care approach that includes shared or supported decisionmaking.”
From Time for united action on depression: a Lancet–World Psychiatric Association Commission (2022) · cites this paper as [601]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · cited 1×
“For studies in which the authors had not defined a primary outcome, we defined the primary outcome as the one that had been used for the sample size calculation [25].”
MethodSelf-citationFrom Triple to Quadruple Aim: Care of the Patient Requires Care of the Provider
Bodenheimer, T., & Sinsky, C. (2014) · The Annals of Family Medicine · doi:10.1370/afm.1713 · Cited by 3,462
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., Whitty, P., Eccles, M., Matowe, L., Shirran, L., et al. (2004) · Health Technology Assessment · doi:10.3310/hta8060 · Cited by 3,044
How should meta‐regression analyses be undertaken and interpreted?
Thompson, S. G., & Higgins, J. P. T. (2002) · Statistics in Medicine · doi:10.1002/sim.1187 · Cited by 3,026
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.017 · Cited by 2,320
Forsetlund, L., Bjørndal, A., Rashidian, A., Jamtvedt, G., O'Brien, M. A., Wolf, F. M., Davis, D., Odgaard-Jensen, J., & Oxman, A. D. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003030.pub2 · Cited by 1,849
Educational outreach visits: effects on professional practice and health care outcomes
O'Brien, M. A., Rogers, S., Jamtvedt, G., Oxman, A. D., Odgaard-Jensen, J., Kristoffersen, D. T., Forsetlund, L., Bainbridge, D., Freemantle, N., Davis, D., et al. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000409.pub2 · Cited by 1,279
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J. M., Thomas, R. E., MacLennan, G., Fraser, C., Ramsay, C. R., Vale, L., Whitty, P., Eccles, M. P., Matowe, L., Shirran, L., et al. (2005) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462305290190 · Cited by 864
Research works printed but not linked: 2
- Fretheim A, Flottorp S, Oxman A. Effect of interventions for implementing clinical practice guidelines. Oslo: Knowledge Centre for the Health Services at The Norwegian Institute of Publich Health (NIPH); 2015. Report from Norwegian Knowledge Centre for the Health Services (NOKC) No. 10-2015. Available from: https://www.ncbi.nlm.nih.gov/books/NBK390571/ .
- EPOC resources for review authors. http://epoc.cochrane.org/resources/epoc-resources-review-authors . Accessed Jan 2016.
Cited by
Sorted by citations
Items 101–156 of 156
Flemington, T., Hort, J., Marks, S., Tankel, A. S., Tzioumi, D., & Fraser, J. (2025) · Global Implementation Research and Applications · doi:10.1007/s43477-025-00175-5 · Cited by 2
Walraven, J. E. W., Verhoeven, R. H. A., van der Hoeven, J. J. M., van der Meulen, R., Lemmens, V. E. P. P., Hesselink, G., & Desar, I. M. E. (2023) · Frontiers in Oncology · doi:10.3389/fonc.2023.1178165 · Cited by 2
Parra-Calderón, C. L., Román-Villarán, E., Álvarez-Romero, C., Escobar-Rodríguez, G. A., Martínez‐Brocca, M. A., Martínez-García, A., García-García, J. A., & Escalona-Cuaresma, M. J. (2023) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2023.105208 · Cited by 2
Systems Design Framework for a Practice-Based Evidence Approached Clinical Decision Support Systems
Osop, H., & Sahama, T. R. (2019) · Australasian Computer Science Week Multiconference (ACSW) · doi:10.1145/3290688.3290742 · Cited by 1
Hornick, M. M., Talwar, A. A., Voytik, M., Gala, Z. S., Amro, C., Khan, S., Deguchi, A. N., Dao, E., Khodyakov, D., Rendle, K. A. S., et al. (2025) · Journal of Surgical Research · doi:10.1016/j.jss.2025.09.089 · Cited by 1
Solli, R., Olsen, N. R., Kvæl, L. A. H., de Velde, S. V., Pripp, A. H., Flottorp, S. A., & Brovold, T. (2026) · PLoS ONE · doi:10.1371/journal.pone.0340025 · Cited by 1
Self-citationMiner, D. S., Knapp, G., Lambert, K., Brabble, K., Dennis, C. R., Markantonis, J., Pierce, J., Hanna, J. J., & Medford, R. J. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag003 · Cited by 1
Waldock, W. J., Gilchrist, M. J., Ashrafian, H., Darzi, A. W., & Franklin, B. D. (2026) · npj Antimicrobials and Resistance · doi:10.1038/s44259-026-00187-7 · Cited by 1
Improving Healthcare Workforce and Patient Safety: A Scoping Meta-Review
Braun, B. I., Bozikis, M. R., Casey, M., Weissman, D. N., Orze, K. J., Riehle, A. I., & DeJoy, D. M. (2026) · Workplace Health & Safety · doi:10.1177/21650799261446613 · Cited by 1
U. S. gas pipelines preparing for life under FERC order 636
Koen, A. D. (1992) · Oil & gas journal/Oil and gas journal · Cited by 1
Testing the Applicability of Digital Decision Support on a Nationwide EHR
Metsallik, J., & Ross, P. (2021) · International Conference on Digital Economy · doi:10.1007/978-3-030-92909-1_9 · Cited by 1
The Integration of Clinical Decision Support Systems Into Pediatric Subspecialty Care
Hosoda-Urban, T., Counts, N. Z., Lois, B. H., & Risser, H. J. (2021) · Clinical Practice in Pediatric Psychology · doi:10.1037/cpp0000396 · Cited by 1
Kherabi, Y., Ming, D. K. Y., Rawson, T. M., & Peiffer‐Smadja, N. (2022) · Advances in medical technologies and clinical practice book series · doi:10.4018/978-1-6684-5092-5.ch007 · Cited by 1
Vanegas, M. A. Z. (2019) · Revista de Salud Pública · doi:10.15446/rsap.v21n2.75062 · Cited by 1
Marco-Ruiz, L., Malm-Nicolaisen, K., Makhlysheva, A., & Pedersen, R. (2020) · International Conference on eHealth, Telemedicine, and Social Medicine · Cited by 1
Self-citationManski-Nankervis, J.-A. E., Hunter, B., Lumsden, N. G., Laughlin, A., McMorrow, R., Boyle, D. I., Chondros, P., Jesudason, S., Radford, J. C., Prictor, M., et al. (2024) · JMIR Formative Research · doi:10.2196/54147 · Cited by 1
Leitch, S. V., Zeng, J., Smith, A. J., & Stokes, T. (2020) · Research Square · doi:10.21203/rs.3.rs-134925/v1 · Cited by 2
Editorial: Guidelines - innovation needed to overcome barriers to use
Manski-Nankervis, J.-A. E. (2022) · Australian Prescriber · doi:10.18773/austprescr.2022.027 · Cited by 1
Health System–Community Partnerships: Implementation and Intensity Matter
Liss, D. T., & Persell, S. D. (2020) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2019.09.021
Hong, D., Shi, W., Lu, X., Jiaying, W., Lou, Y., & Li, L. (2020) · medRxiv · doi:10.1101/2020.01.29.20019364
How to Help Clients Make Effective Decisions About Romantic Relationships: A Systematic ReviewPreprint
Beddoe, L., Kučuk, I., & Noetel, M. (2022) · doi:10.31234/osf.io/xgru7
Cognitive considerations for health information technology in clinical team environments
Franklin, A., & Zhang, J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00024-3
Evaluation of clinical decision support
Benson, N. M., Salmasian, H., & Bates, D. W. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00026-7
Janiak, E., Rexrode, K. M., Santacroce, L. M., Johns, S. L., Behn, M., Braaten, K. P., & Feldman, C. H. (2024) · PLoS ONE · doi:10.1371/journal.pone.0303930
THE IMPLEMENTATION OF ELECTRONIC-BASED SYSTEM IN REDUCTION OF MEDICATION ERROR: A LITERATURE REVIEW
Saho, M., Hariyati, R. T. S., & Pujasari, H. (2024) · Cendekia Medika Jurnal Stikes Al-Ma`arif Baturaja · doi:10.52235/cendekiamedika.v9i2.381
A Measurement Science Framework to Optimize CDS for Opioid Use Disorder Treatment in the ED
Iscoe, M. S., Hooper, C. D., Levy, D. R., Lutz, J., Paek, H., Rose, C. C., Kannampallil, T., Meeker, D., Dziura, J. D., & Melnick, E. R. (2025) · Applied Clinical Informatics · doi:10.1055/a-2595-0317
Söling, S., Köberlein-Neu, J., Müller, B. S., Dinh, T. S., Muth, C., Pfaff, H., & Karbach, U. (2020) · Figshare · doi:10.6084/m9.figshare.12986844
Ullah, A., William, W., Bradley, F., Nelson, P. A., Dowding, D., Morciano, M., & Cullum, N. (2024) · Figshare · doi:10.6084/m9.figshare.26563540
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600119
Chen, W., O’Bryan, C. M., Gorham, G., Howard, K., Balasubramanya, B., Coffey, P., Abeyaratne, A., & Cass, A. (2022) · Figshare · doi:10.6084/m9.figshare.20401242
Lin, L., Alam, P., Fearon, E., & Hargreaves, J. R. (2020) · Figshare · doi:10.6084/m9.figshare.13120782
Akhloufi, H., van der Sijs, H., Melles, D. C., van der Hoeven, C. P., Vogel, M., Mouton, J. W., & Verbon, A. (2022) · Figshare · doi:10.6084/m9.figshare.19745652
Lin, L., Alam, P., Fearon, E., & Hargreaves, J. R. (2020) · Figshare · doi:10.6084/m9.figshare.13120785
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600116
Chen, W., O’Bryan, C. M., Gorham, G., Howard, K., Balasubramanya, B., Coffey, P., Abeyaratne, A., & Cass, A. (2022) · Figshare · doi:10.6084/m9.figshare.20401254
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600113
Söling, S., Köberlein-Neu, J., Müller, B. S., Dinh, T. S., Muth, C., Pfaff, H., & Karbach, U. (2020) · Figshare · doi:10.6084/m9.figshare.12986847
Lin, L., Alam, P., Fearon, E., & Hargreaves, J. R. (2020) · Figshare · doi:10.6084/m9.figshare.13120776
Chen, W. T., O’Bryan, C. M., Gorham, G., Howard, K., Balasubramanya, B., Coffey, P., Abeyaratne, A., & Cass, A. (2022) · Figshare · doi:10.6084/m9.figshare.20401239
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600107
Chen, W., O’Bryan, C. M., Gorham, G., Howard, K., Balasubramanya, B., Coffey, P., Abeyaratne, A., & Cass, A. (2022) · Figshare · doi:10.6084/m9.figshare.20401245
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600104
Söling, S., Köberlein-Neu, J., Müller, B. S., Dinh, T. S., Muth, C., Pfaff, H., & Karbach, U. (2020) · Figshare · doi:10.6084/m9.figshare.12986850
Lin, L., Alam, P., Fearon, E., & Hargreaves, J. R. (2020) · Figshare · doi:10.6084/m9.figshare.13120779
Söling, S., Demirer, I., Köberlein-Neu, J., Hower, K. I., Müller, B. S., Pfaff, H., & Karbach, U. (2024) · Figshare · doi:10.6084/m9.figshare.26600110
Saman, D. M., Chrenka, E. A., Harry, M. L., Allen, C. I., Freitag, L. A., Asche, S. E., Truitt, A. R., Ekstrom, H. L., Pj, O., Sperl-Hillen, J. M., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14820775
Chen, W., O’Bryan, C. M., Gorham, G., Howard, K., Balasubramanya, B., Coffey, P., Abeyaratne, A., & Cass, A. (2022) · Figshare · doi:10.6084/m9.figshare.20401251
Nemorin, A., Norton, D. T., Green, C. V., Jerry, M. S., Wurcel, A. G., & Blumenthal, K. G. (2025) · Journal of Allergy and Clinical Immunology Global · doi:10.1016/j.jacig.2025.100636
Clinician-Personalized Clinical Decision Support Tools: The Right Nudge for the Right (Preprint)Preprint
Trinkley, K. E., Gurfinkel, D., Allen, L. A., Bennett, T. D., Gatto, A. J., Glasgow, R. E., Huebschmann, A. G., Lin, C., Malone, D. C., Matlock, D. D., et al. (2026) · doi:10.2196/preprints.105384
Factors associated with chronic pain clinical decision support use in primary care
McCord, E., Apathy, N. C., Blackburn, J. L., Holmes, A. M., Sanner, L. M., Harle, C. A., & Mazurenko, O. (2026) · PLOS Digital Health · doi:10.1371/journal.pdig.0001032
Kurtzman, R., Ozkaynak, M., Jiménez, F. N., Kukhareva, P., Nwefo, R., Desai, P. J., Workgroup, C. M. A. O., Kurtzman, R., Ozkaynak, M., Jiménez, F. N., et al. (2026) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-026-03715-7
Lalani, B., Abusamaan, M. S., Demidowich, A. P., Zilbermint, M. F., Sarkar, S., & Mathioudakis, N. N. (2026) · Journal of Diabetes Science and Technology · doi:10.1177/19322968261468379
McCord, E., Blackburn, J. L., Holmes, A. M., Apathy, N. C., Grout, R. W., Harle, C. A., & Mazurenko, O. (2026) · medRxiv · doi:10.64898/2026.10.07.26364970
Vasiliou, V. S., Ewing, C., Martins-Martinho, J., Lopes, A. R., Mackie, S. L., Ponte, C., & Luqmani, R. A. (2026) · Research Square · doi:10.21203/rs.3.rs-10606185/v1
Bye, E. M., Anstett, T. J., Sciarra, B., Mitchell, M., Stenehjem, E. A., & Frasca, K. (2026) · HIV Medicine · doi:10.1111/hiv.70336
Metadata
- Authors
- Van de Velde, S., Heselmans, A., Delvaux, N., Brandt, L., Marco-Ruiz, L., Spitaels, D., Cloetens, H., Kortteisto, T., Roshanov, P., Kunnamo, I., Aertgeerts, B., Vandvik, P. O., & Flottorp, S.
- Affiliations
- KU Leuven (BE)
- University of Oslo (NO)
- Tampere University Hospital (FI)
- McMaster University (CA)
- Finnish Medical Society Duodecim (FI)
- Norwegian Institute of Public Health (NO)
- ORCID
- Venue
- Implementation Science
- Published
- 2018-08-20
- Volume
- 13
- Issue
- 1
- Pages
- 114-114
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC6102833
- OpenAlex
- W2888668484
- Cited by
- 156
- References
- 40
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 14 · 2023: 18 · 2022: 26 · 2021: 20 · 2020: 35 · 2019: 15 · 2018: 1
- FWCI
- 3.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- BIBSYS Brage (BIBSYS (Norway)) (acceptedVersion, cc-by-nc-sa)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by-nc-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- Horizon 2020 (654981)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- computerized clinical decision support, success factors, patient satisfaction, economic outcomes, learning healthcare systems
- MeSH terms
- Decision Support Techniques, Humans, Quality of Life, Randomized Controlled Trials as Topic, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
