The DOI matches its Crossref record; 40 of the 40 references it cites were found in a registry or an index.
A systematic review of trials evaluating success factors of interventions with computerised clinical decision support
Open accessAbstract
BACKGROUND: Computerised clinical decision support (CDS) can potentially better inform decisions, and it can help with the management of information overload. It is perceived to be a key component of a learning health care system. Despite its increasing implementation worldwide, it remains uncertain why the effect of CDS varies and which factors make CDS more effective. OBJECTIVE: To examine which factors make CDS strategies more effective on a number of outcomes, including adherence to recommended practice, patient outcome measures, economic measures, provider or patient satisfaction, and medical decision quality. METHODS: We identified randomised controlled trials, non-randomised trials, and controlled before-and-after studies that directly compared CDS implementation with a given factor to CDS without that factor by searching CENTRAL, MEDLINE, EMBASE, and CINAHL and checking reference lists of relevant studies. We considered CDS with any objective for any condition in any healthcare setting. We included CDS interventions that were either displayed on screen or provided on paper and that were directed at healthcare professionals or targeted at both professionals and patients. The reviewers screened the potentially relevant studies in duplicate. They extracted data and assessed risk of bias in independent pairs or individually followed by a double check by another reviewer. We summarised results using medians and interquartile ranges and rated our certainty in the evidence using the GRADE system. RESULTS: We identified 66 head-to-head trials that we synthesised across 14 comparisons of CDS intervention factors. Providing CDS automatically versus on demand led to large improvements in adherence. Displaying CDS on-screen versus on paper led to moderate improvements and making CDS more versus less patient-specific improved adherence modestly. When CDS interventions were combined with professional-oriented strategies, combined with patient-oriented strategies, or combined with staff-oriented strategies, then adherence improved slightly. Providing CDS to patients slightly increased adherence versus CDS aimed at the healthcare provider only. Making CDS advice more explicit and requiring users to respond to the advice made little or no difference. The CDS intervention factors made little or no difference to patient outcomes. The results for economic outcomes and satisfaction outcomes were sparse. CONCLUSION: Multiple factors may affect the success of CDS interventions. CDS may be more effective when the advice is provided automatically and displayed on-screen and when the suggestions are more patient-specific. CDS interventions combined with other strategies probably also improves adherence. Providing CDS directly to patients may also positively affect adherence. The certainty of the evidence was low to moderate for all factors. TRIAL REGISTRATION: PROSPERO, CRD42016033738.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Although previous studies have studied factors affecting the implementation of CDS tools [48], the use of black box models in health care settings presents unique challenges and opportunities related to trust and transparency.”
From Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study (2020) · cites this paper as [48]
In the citing paper
“The WPA in common with WHO advocates for a pragmatic approach to implementing viable alternatives to coercion, 601 one of which is adopting a community-based collaborative care approach that includes shared or supported decisionmaking.”
From Time for united action on depression: a Lancet–World Psychiatric Association Commission (2022) · cites this paper as [601]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
GRADE guidelines: 3. Rating the quality of evidence
Balshem, H., Helfand, M., Schünemann, H. J., Oxman, A. D., Kunz, R., Brozek, J., Vist, G. E., Falck-Ytter, Y., Meerpohl, J., & Norris, S. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.015 · Cited by 8,285
Audit and feedback: effects on professional practice and healthcare outcomes
Ivers, N., Jamtvedt, G., Flottorp, S., Young, J. M., Odgaard-Jensen, J., French, S. D., O'Brien, M. A., Johansen, M., Grimshaw, J., & Oxman, A. D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000259.pub3 · Cited by 5,485 · cited 1×
“For studies in which the authors had not defined a primary outcome, we defined the primary outcome as the one that had been used for the sample size calculation [25].”
MethodSelf-citationFrom Triple to Quadruple Aim: Care of the Patient Requires Care of the Provider
Bodenheimer, T., & Sinsky, C. (2014) · The Annals of Family Medicine · doi:10.1370/afm.1713 · Cited by 3,462
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J., Thomas, R., MacLennan, G., Fraser, C., Ramsay, C., Vale, L., Whitty, P., Eccles, M., Matowe, L., Shirran, L., et al. (2004) · Health Technology Assessment · doi:10.3310/hta8060 · Cited by 3,044
How should meta‐regression analyses be undertaken and interpreted?
Thompson, S. G., & Higgins, J. P. T. (2002) · Statistics in Medicine · doi:10.1002/sim.1187 · Cited by 3,026
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703
GRADE guidelines: 7. Rating the quality of evidence—inconsistency
Guyatt, G. H., Oxman, A. D., Kunz, R., Woodcock, J., Brozek, J., Helfand, M., Alonso-Coello, P., Glasziou, P., Jaeschke, R., Akl, E. A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.03.017 · Cited by 2,320
Forsetlund, L., Bjørndal, A., Rashidian, A., Jamtvedt, G., O'Brien, M. A., Wolf, F. M., Davis, D., Odgaard-Jensen, J., & Oxman, A. D. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd003030.pub2 · Cited by 1,849
Educational outreach visits: effects on professional practice and health care outcomes
O'Brien, M. A., Rogers, S., Jamtvedt, G., Oxman, A. D., Odgaard-Jensen, J., Kristoffersen, D. T., Forsetlund, L., Bainbridge, D., Freemantle, N., Davis, D., et al. (2007) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd000409.pub2 · Cited by 1,279
Effectiveness and efficiency of guideline dissemination and implementation strategies
Grimshaw, J. M., Thomas, R. E., MacLennan, G., Fraser, C., Ramsay, C. R., Vale, L., Whitty, P., Eccles, M. P., Matowe, L., Shirran, L., et al. (2005) · International Journal of Technology Assessment in Health Care · doi:10.1017/s0266462305290190 · Cited by 864
The Quadruple Aim: care, health, cost and meaning in work
Sikka, R., Morath, J. M., & Leape, L. L. (2015) · BMJ Quality & Safety · doi:10.1136/bmjqs-2015-004160 · Cited by 691
The Learning Healthcare System: Workshop Summary
Olsen, L., Aisner, D. L., & McGinnis, J. M. (2013) · Cited by 588
Jaspers, M. W. M., Smeulers, M., Vermeulen, H., & Peute, L. W. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000094 · Cited by 525
The effects of on-screen, point of care computer reminders on processes and outcomes of care
Shojania, K. G., Jennings, A., Ramsay, C. R., Grimshaw, J., Kwan, J. L., & Lo, L. (2009) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001096.pub2 · Cited by 484
Roshanov, P. S., Fernandes, N., Wilczynski, J. M., Hemens, B. J., You, J. J., Handler, S. M., Nieuwlaat, R., Souza, N. M., Beyene, J., Spall, H. G. C. V., et al. (2013) · BMJ · doi:10.1136/bmj.f657 · Cited by 470
Self-citationMoja, L., Kwag, K. H., Lytras, T., Bertizzolo, L., Brandt, L., Pecoraro, V. L., Rigon, G., Vaona, A., Ruggiero, F. M., Mangia, M., et al. (2014) · American Journal of Public Health · doi:10.2105/ajph.2014.302164 · Cited by 341
Self-citationNuckols, T. K., Smith-Spangler, C., Morton, S. C., Asch, S. M., Patel, V. M., Anderson, L. J., Deichsel, E. L., & Shekelle, P. G. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-56 · Cited by 340
Some unintended consequences of clinical decision support systems.
Ash, J. S., Sittig, D. F., Campbell, E. M., Guappone, K. P., & Dykstra, R. H. (2007) · PubMed Central · Cited by 298
Clinical Decision Support: a 25 Year Retrospective and a 25 Year Vision
Sittig, D. F., Wright, A. T., & Middleton, B. (2016) · Yearbook of Medical Informatics · doi:10.15265/iys-2016-s034 · Cited by 287
PROSPERO International prospective register of systematic reviews
Quayle, E., Cooper, K., Newman, E., & Cariola, L. A. (2017) · Cited by 186
Enabling health care decisionmaking through clinical decision support and knowledge management.
Lobach, D. F., Sanders, G. D., Bright, T. J., Wong, A. F., Ravi, D., Bristow, E., Bastian, L. A., Coeytaux, R. R., Samsa, G. P., Hasselblad, V., et al. (2012) · PubMed Central · Cited by 179
Arditi, C., Rège-Walther, M., Wyatt, J. C., Durieux, P., & Burnand, B. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001175.pub3 · Cited by 179
panel, G. E., de Velde, S. V., Kunnamo, I., Roshanov, P. S., Kortteisto, T., Aertgeerts, B., Vandvik, P. O., & Flottorp, S. A. (2018) · Implementation Science · doi:10.1186/s13012-018-0772-3 · Cited by 136
Self-citationPharmaceutical policies: effects of financial incentives for prescribers
Rashidian, A., Omidvari, A., Vali, Y., Sturm, H., & Oxman, A. D. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd006731.pub2 · Cited by 130
The effectiveness of computerized clinical guidelines in the process of care: a systematic review
Damiani, G., Pinnarelli, L., Colosimo, S. C., Almiento, R., Sicuro, L., Galasso, R., Sommella, L., & Ricciardi, W. (2010) · BMC Health Services Research · doi:10.1186/1472-6963-10-2 · Cited by 129
Arditi, C., Rège-Walther, M., Durieux, P., & Burnand, B. (2017) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001175.pub4 · Cited by 106
Strategies for coping with information overload
Smith, R. W. (2010) · BMJ · doi:10.1136/bmj.c7126 · Cited by 75
Effectiveness of Topic-specific Infobuttons: A Randomized Controlled Trial
Fiol, G. D., Haug, P. J., Cimino, J. J., Narus, S. P., Norlin, C., & Mitchell, J. A. (2008) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2725 · Cited by 75
Rosenbloom, S. T., Geissbuhler, A. J., Dupont, W. D., Giuse, D. A., Talbert, D. A., Tierney, W. M., Plummer, W. D., Stead, W. W., & Miller, R. A. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1627 · Cited by 71
Myers, R. B., Jones, S. L., & Sittig, D. F. (2011) · Applied Clinical Informatics · doi:10.4338/aci-2010-11-ra-0064 · Cited by 67
Denig, P., Schuling, J., Haaijer‐Ruskamp, F. M., & Voorham, J. (2014) · BMJ · doi:10.1136/bmj.g5651 · Cited by 64
Tomás Pantoja, C., Grimshaw, J., Colomer, N., Castañón, C., & Léniz, J. (2019) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001174.pub4 · Cited by 57
Sintchenko, V., Magrabi, F., & Tipper, S. (2007) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230701447701 · Cited by 47
Still no magic bullets: pursuing more rigorous research in quality improvement
Shojania, K. G., & Grimshaw, J. M. (2004) · The American Journal of Medicine · doi:10.1016/j.amjmed.2004.03.003 · Cited by 43
The road to effective clinical decision support: are we there yet?
Lobach, D. F. (2013) · BMJ · doi:10.1136/bmj.f1616 · Cited by 27
Brandt, L., Vandvik, P. O., Alonso‐Coello, P., Akl, E. A., Thornton, J., Comas, D. R., Adams, K., O'Connor, P., Guyatt, G. H., & Kristiansen, A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-011569 · Cited by 25
Self-citationManual paper reminders: effects on professional practice and health care outcomes
Tomás Pantoja, C., Green, M., Grimshaw, J., Denig, P., Durieux, P., Gill, P. S., Colomer, N., Castañón, C., & Léniz, J. (2014) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd001174.pub3 · Cited by 18
Addressing the Unit of Analysis in Medical Care Studies
Calhoun, A. W., Guyatt, G. H., Cabana, M. D., Lu, D., Turner, D. A., Valentine, S. L., & Randolph, A. G. (2008) · Medical Care · doi:10.1097/mlr.0b013e3181649412 · Cited by 17
Lobach, D. F., Kawamoto, K., Anstrom, K. J., Silvey, G. M., Willis, J. M., Johnson, F. S., Edwards, R., Simo, J., Phillips, P., Crosslin, D. R., et al. (2013) · Journal of Medical Systems · doi:10.1007/s10916-012-9922-3 · Cited by 12
The perils of meta-regression to identify clinical decision support system success factors
Fillmore, C. L., Rommel, C. A., Welch, B. M., Zhang, M., & Kawamoto, K. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.007 · Cited by 10
Research works printed but not linked: 2
- Fretheim A, Flottorp S, Oxman A. Effect of interventions for implementing clinical practice guidelines. Oslo: Knowledge Centre for the Health Services at The Norwegian Institute of Publich Health (NIPH); 2015. Report from Norwegian Knowledge Centre for the Health Services (NOKC) No. 10-2015. Available from: https://www.ncbi.nlm.nih.gov/books/NBK390571/ .
- EPOC resources for review authors. http://epoc.cochrane.org/resources/epoc-resources-review-authors . Accessed Jan 2016.
Cited by
Sorted by citations
Time for united action on depression: a Lancet–World Psychiatric Association Commission
Herrman, H., Patel, V., Kieling, C., Berk, M., Buchweitz, C., Cuijpers, P., Furukawa, T. A., Kessler, R. C., Kohrt, B. A., Maj, M., et al. (2022) · The Lancet · doi:10.1016/s0140-6736(21)02141-3 · Cited by 1,104
Kwan, J. L., Lo, L., Ferguson, J., Goldberg, H. R., Díaz-Martínez, J. P., TOMLINSON, G. A., Grimshaw, J., & Shojania, K. G. (2020) · BMJ · doi:10.1136/bmj.m3216 · Cited by 493
Jacobs, M., He, J., F. Pradier, M., Lam, B., Ahn, A. C., McCoy, T. H., Perlis, R. H., Doshi-Velez, F., & Gajos, K. Z. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445385 · Cited by 177
Integrating a Machine Learning System Into Clinical Workflows: Qualitative Study
Sandhu, S., Lin, A. L., Brajer, N., Sperling, J., Ratliff, W., Bedoya, A. D., Balu, S., O'Brien, C., & Sendak, M. P. (2020) · Journal of Medical Internet Research · doi:10.2196/22421 · Cited by 168
Chomutare, T., Tejedor, M., Svenning, T. O., Marco-Ruiz, L., Tayefi, M., Lind, K., Godtliebsen, F., Moen, A., Ismail, L., Makhlysheva, A., et al. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph192316359 · Cited by 164
Self-citation
Metadata
- Authors
- Van de Velde, S., Heselmans, A., Delvaux, N., Brandt, L., Marco-Ruiz, L., Spitaels, D., Cloetens, H., Kortteisto, T., Roshanov, P., Kunnamo, I., Aertgeerts, B., Vandvik, P. O., & Flottorp, S.
- Affiliations
- KU Leuven (BE)
- University of Oslo (NO)
- Tampere University Hospital (FI)
- McMaster University (CA)
- Finnish Medical Society Duodecim (FI)
- Norwegian Institute of Public Health (NO)
- ORCID
- Venue
- Implementation Science
- Published
- 2018-08-20
- Volume
- 13
- Issue
- 1
- Pages
- 114-114
- ISSN
- 1748-5908 (ISSN-L)
- PMC
- PMC6102833
- OpenAlex
- W2888668484
- Cited by
- 156
- References
- 40
- Citations by year
- 2026: 11 · 2025: 14 · 2024: 14 · 2023: 18 · 2022: 26 · 2021: 20 · 2020: 35 · 2019: 15 · 2018: 1
- FWCI
- 3.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Implementation Science (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- BIBSYS Brage (BIBSYS (Norway)) (acceptedVersion, cc-by-nc-sa)
- Lirias (KU Leuven) (submittedVersion, other-oa)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, other-oa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by-nc-sa)
- NORA - Norwegian Open Research Archives (submittedVersion, cc-by)
- Funders
- Horizon 2020 (654981)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- computerized clinical decision support, success factors, patient satisfaction, economic outcomes, learning healthcare systems
- MeSH terms
- Decision Support Techniques, Humans, Quality of Life, Randomized Controlled Trials as Topic, Decision Support Systems, Clinical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
