A taxonomic description of computer-based clinical decision support systems
Open accessAbstract
Objective Computer-based clinical decision support systems (CDSSs) vary greatly in design and function. Using a taxonomy that we had previously developed, we describe the characteristics of CDSSs reported in the literature.Methods We searched PubMed and the Cochrane Library for randomized controlled trials (RCTs) published in English between 1998 and 2003 that evaluated CDSSs. We coded each CDSS using our taxonomy.Results 58 studies met our inclusion criteria. The 74 reported CDSSs varied greatly in context of use, knowledge and data sources, nature of decision support offered, information delivery, and workflow impact. Two distinct subsets of CDSSs were seen: patient-directed systems that provided decision support for preventive care or health-related behaviors via mail or phone (38% of systems), and inpatient systems targeting clinicians with online decision support and direct online execution of the recommendations (18%). 84% of the CDSSs required extra staffing for handling CDSS-related input or output.Conclusion Reported CDSSs are heterogeneous along many dimensions. Caution should be taken in generalizing the results of CDSS RCTs to different clinical or workflow settings.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
A meta-analysis would have been inappropriate because of significant heterogeneity among studies in sample characteristics, specific technologies studied, and functions of those technologies.
“As others who have reviewed health IT research have pointed out, a meta-analysis would have been inappropriate because of the significant heterogeneity among studies in terms of sample characteristics, specific technologies studied, and functions of those technologies [93, 94].”
From The Technology Acceptance Model: Its past and its future in health care (2010) · cites this paper as [93]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
A Coefficient of Agreement for Nominal Scales
Cohen, J. (1960) · Educational and Psychological Measurement · doi:10.1177/001316446002000104 · Cited by 42,398 · see citing works
Institute of Medicine (2000) · National Academies Press eBooks · doi:10.17226/9728 · Cited by 14,174 · see citing works
Wall, A. (2000) · British Journal of Healthcare Management · doi:10.12968/bjhc.2000.6.9.19311 · Cited by 5,909 · see citing works
Garg, A. X., Adhikari, N. K. J., McDonald, H., Rosas-Arellano, M. P., Devereaux, P. J., Beyene, J., Sam, J., & Haynes, R. B. (2005) · JAMA · doi:10.1001/jama.293.10.1223 · Cited by 2,936 · see citing works
Kawamoto, K., Houlihan, C. A., Balas, E. A., & Lobach, D. F. (2005) · BMJ · doi:10.1136/bmj.38398.500764.8f · Cited by 2,703 · see citing works
Bates, D. W. (1998) · JAMA · doi:10.1001/jama.280.15.1311 · Cited by 2,107 · see citing works
Leavitt, M. (2001) · PubMed · Cited by 1,835 · see citing works
A Computer-Assisted Management Program for Antibiotics and Other Antiinfective Agents
Evans, R. S., Pestotnik, S. L., Classen, D. C., Clemmer, T. P., Weaver, L. K., Orme, J. F., Lloyd, J. F., & Burke, J. P. (1998) · New England Journal of Medicine · doi:10.1056/nejm199801223380406 · Cited by 1,060 · see citing works
Clinical decision support systems for the practice of evidence-based medicine
Sim, I., Gorman, P., Greenes, R. A., Haynes, R. B., Kaplan, B., Lehmann, H., & Tang, P. C. (2001) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2001.0080527 · Cited by 732 · see citing works
Self-citationA COMPUTER-ASSISTED MANAGEMENT PROGRAM FOR ANTIBIOTICS AND OTHER ANTIINFECTIVE AGENTS
Vans, R. S. C. E., Estotnik, S. T. L. P., Lassen, D. A. C. C., Lemmer, T., Eaver, L. I. K. W., Rme, J. A. F. O., Loyd, J. A. F. L., & Urke, J. O. P. B. (1998) · Cited by 692 · see citing works
A Computerized Reminder System to Increase the Use of Preventive Care for Hospitalized Patients
Dexter, P. R., Perkins, S., Overhage, J. M., Maharry, K., Kohler, R. B., & McDonald, C. J. (2001) · New England Journal of Medicine · doi:10.1056/nejmsa010181 · Cited by 519 · see citing works
Eccles, M. (2002) · BMJ · doi:10.1136/bmj.325.7370.941 · Cited by 354 · cited 1× · see citing works
“Eccles [31] 2002 EMR with recommendations for treatment of angina and asthma No recommendations Etter [32] 2001 Mailed, computer-generated, personalized smoking cessation counseling materials No letter”
MethodProchaska, J. O., Velicer, W. F., Fava, J. L., Rossi, J. S., & Tsoh, J. Y. (2001) · Addictive Behaviors · doi:10.1016/s0306-4603(00)00151-9 · Cited by 307 · see citing works
Tamblyn, R. M., Huang, A. R., Perreault, R. B., Jacques, A., Roy, D., Hanley, J. A., McLeod, P. J., & Laprise, R. (2003) · Europe PMC (PubMed Central) · Cited by 298 · see citing works
Bates, D. W., Kuperman, G. J., Rittenberg, E., Teich, J. M., Fiskio, J., Ma’luf, N., Onderdonk, A., Wybenga, D., Winkelman, J., Brennan, T. A., et al. (1999) · The American Journal of Medicine · doi:10.1016/s0002-9343(98)00410-0 · Cited by 292 · see citing works
Montgomery, A. A. (2000) · BMJ · doi:10.1136/bmj.320.7236.686 · Cited by 279 · see citing works
The contextual nature of medical information
Berg, M., & Goorman, E. (1999) · International Journal of Medical Informatics · doi:10.1016/s1386-5056(99)00041-6 · Cited by 253 · see citing works
Kuperman, G. J., Teich, J. M., Tanasijevic, M. J., Ma'Luf, N., Rittenberg, E., Jha, A., Fiskio, J., Winkelman, J., & Bates, D. W. (1999) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1999.0060512 · Cited by 252 · see citing works
Rousseau, N. S., McColl, E., Newton, J., Grimshaw, J., & Eccles, M. (2003) · BMJ · doi:10.1136/bmj.326.7384.314 · Cited by 221 · see citing works
Multicentre randomised study of computerised anticoagulant dosage
Poller, L., Shiach, C. R., MacCallum, P. K., Johansen, A., Münster, A. B., Magalhäes, A., & Jespersen, J. (1998) · The Lancet · doi:10.1016/s0140-6736(98)04147-6 · Cited by 221 · see citing works
Reducing Vancomycin Use Utilizing a Computer Guideline: Results of a Randomized Controlled Trial
Shojania, K. G., Yokoe, D. S., Platt, R. G., Fiskio, J. M., Ma'Luf, N., & Bates, D. W. (1998) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1998.0050554 · Cited by 211 · see citing works
Demakis, J. G. (2000) · JAMA · doi:10.1001/jama.284.11.1411 · Cited by 206 · see citing works
Effects of computerized guidelines for managing heart disease in primary care
Tierney, W. M., Overhage, J. M., Murray, M. D., Harris, L. E., Zhou, X., Eckert, G. J., Smith, F. E., Nienaber, N. A., McDonald, C. J., & Wolinsky, F. D. (2003) · Journal of General Internal Medicine · doi:10.1111/j.1525-1497.2003.30635.x · Cited by 199 · see citing works
Dexter, P. R., Wolinsky, F. D., Gramelspacher, G. P., Zhou, X., Eckert, G. J., Waisburd, M., & Tierney, W. M. (1998) · Annals of Internal Medicine · doi:10.7326/0003-4819-128-2-199801150-00005 · Cited by 190 · see citing works
Home computers and Internet use in the United States, August 2000
Newburger, E. C. (2001) · Cited by 184 · see citing works
Fitzmaurice, D. A., Hobbs, R. F., Murray, E. T., HOLDER, R. L., Allan, T. F., & Rose, P. E. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.15.2343 · Cited by 174 · see citing works
Molenaar, S., Sprangers, M. A. G., Rutgers, E. J. T., Luiten, E. J. T., Mulder, J., Bossuyt, P. M., van Everdingen, J. J., Oosterveld, P., & de Haes, J. C. J. M. (2001) · Journal of Clinical Oncology · doi:10.1200/jco.2001.19.6.1676 · Cited by 170 · see citing works
Gross, T. M., Kayne, D. M., King, A. B., Rother, C., & Juth, S. E. (2003) · Diabetes Technology & Therapeutics · doi:10.1089/152091503765691848 · Cited by 144 · see citing works
Effectiveness of a Computer-Tailored Smoking Cessation Program
Etter, J., & Perneger, T. (2001) · Archives of Internal Medicine · doi:10.1001/archinte.161.21.2596 · Cited by 131 · see citing works
Rollman, B. L., Hanusa, B. H., Lowe, H., Gilbert, T., Kapoor, W. N., & Schulberg, H. C. (2002) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.2002.10421.x · Cited by 124 · see citing works
Can tailored interventions increase mammography use among HMO women?
Lipkus, I. M., Rimer, B. K., Halabi, S., & Strigo, T. S. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00106-3 · Cited by 121 · see citing works
Christakis, D., Zimmerman, F. J., Wright, J. A., Garrison, M. M., Rivara, F. P., & Davis, R. L. (2001) · PEDIATRICS · doi:10.1542/peds.107.2.e15 · Cited by 118 · see citing works
Use of an on-line pager system to increase adherence to antiretroviral medications
Safren, S. A., Hendriksen, E. S., Desousa, N., Boswell, S. L., & Mayer, K. H. (2003) · AIDS Care · doi:10.1080/09540120310001618630 · Cited by 110 · see citing works
Manotti, C., Moia, M., Palareti, G., et al. (2001) · PubMed · Cited by 103 · see citing works
Targeting Smokers with Low Readiness to Change with Tailored and Nontailored Self-Help Materials
Dijkstra, A., de Vries, H., & Roijackers, J. (1999) · Preventive Medicine · doi:10.1006/pmed.1998.0407 · Cited by 100 · see citing works
McKinley, B. A., Moore, F. A., Sailors, R. M., Cocanour, C. S., Marquez, A., Wright, R. K., Tonnesen, A. S., Wallace, C. J., Morris, A. H., & East, T. D. (2001) · doi:10.1097/00005373-200103000-00004 · Cited by 100 · see citing works
Cannon, D. S., & Allen, S. N. (2000) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2000.0070196 · Cited by 99 · see citing works
Assessment of Decision Support for Blood Test Ordering in Primary Care
van Wijk, M. A. M., van der Lei, J., Mosseveld, M., Bohnen, A. M., & van Bemmel, J. H. (2001) · Annals of Internal Medicine · doi:10.7326/0003-4819-134-4-200102200-00010 · Cited by 98 · see citing works
Lieu, T. A., Capra, A. M., Makol, J., BLACK, S. B., Shinefield, H. R., & for the Immunization Message Study Group (1998) · PEDIATRICS · doi:10.1542/peds.101.4.e3 · Cited by 95 · see citing works
Informed consent in the childrens cancer group
Childrens Cancer Group, Kodish, E. D., Pentz, R. D., Noll, R. B., Ruccione, K., Buckley, J. D., & Lange, B. J. (1998) · Cancer · doi:10.1002/(sici)1097-0142(19980615)82:12 · Cited by 91 · see citing works
Filippi, A., Sabatini, A., Badioli, L., Samani, F., Mazzaglia, G., Catapano, A. L., & Cricelli, C. A. (2003) · Diabetes Care · doi:10.2337/diacare.26.5.1497 · Cited by 91 · see citing works
McCowan, C. (2001) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230110067890 · Cited by 90 · see citing works
Shiffman, R. N., Freudigman, M. ‡. K. A., Brandt, C., Liaw, Y., & Navedo, D. D. (2000) · PEDIATRICS · doi:10.1542/peds.105.4.767 · Cited by 87 · see citing works
Innovative Newsletter Interventions Improve Fruit and Vegetable Consumption in Healthy Adults
LUTZ, S. F., AMMERMAN, A. S., ATWOOD, J. R., CAMPBELL, M. K., DeVELLIS, R. F., & ROSAMOND, W. D. (1999) · Journal of the American Dietetic Association · doi:10.1016/s0002-8223(99)00169-8 · Cited by 86 · see citing works
Lennox, A. S., Osman, L. M., Reiter, E., Robertson, R., Friend, J., McCann, I., Skatun, D., & Donnan, P. T. (2001) · BMJ · doi:10.1136/bmj.322.7299.1396 · Cited by 82 · see citing works
Schriger, D. L., Gibbons, P. S., Langone, C. A., Lee, S. D., & Altshuler, L. L. (2001) · Annals of Emergency Medicine · doi:10.1067/mem.2001.112255 · Cited by 82 · see citing works
Branston, L., Greening, S. E., Newcombe, R. G., et al. (2002) · European Journal of Cancer · doi:10.1016/s0959-8049(01)00258-1 · Cited by 79 · see citing works
Shiffman, S. S., Paty, J. A., Rohay, J. M., Marino, M. E. D., & Gitchell, J. G. (2000) · Archives of Internal Medicine · doi:10.1001/archinte.160.11.1675 · Cited by 76 · see citing works
The impact of computer-generated messages on childhood immunization coverage
Dini, E. F., Linkins, R. W., & Sigafoos, J. (2000) · American Journal of Preventive Medicine · doi:10.1016/s0749-3797(99)00086-0 · Cited by 70 · see citing works
Zanetti, G., Flanagan, H. L., Cohn, L. H., et al. (2003) · Infection Control and Hospital Epidemiology · doi:10.1086/502109 · Cited by 65 · see citing works
Selker, H. P., Beshansky, J. R., Griffith, J. L., & for the TPI Trial Investigators* (2002) · Annals of Internal Medicine · doi:10.7326/0003-4819-137-2-200207160-00006 · Cited by 63 · see citing works
Ageno, W., & Turpie, A. G. G. (1998) · Thrombosis Research · doi:10.1016/s0049-3848(98)00092-9 · Cited by 59 · see citing works
Williams, R. B. (1998) · Archives of Family Medicine · doi:10.1001/archfami.7.4.338 · Cited by 58 · see citing works
Johnson, J., Nowacki, B., Turpie, A., & Ageno, W. (2000) · Thrombosis and Haemostasis · doi:10.1055/s-0037-1613932 · Cited by 58 · see citing works
Influenza immunization in a managed care organization
Baker, A. M., McCarthy, B. D., Gurley, V. F., & Yood, M. U. (1998) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1998.00136.x · Cited by 50 · see citing works
Nieminen, P., Hakama, M., Viikki, M., et al. (2002) · International Journal of Cancer · doi:10.1002/ijc.10839 · Cited by 43 · see citing works
Prescription in Ischaemic Stroke Management (PRISM) Study Group (2003) · QJM · doi:10.1093/qjmed/hcg019 · Cited by 34 · see citing works
Using an interactive video disk in breast cancer patient support.
Maslin, A., Baum, M., Walker, J. S., A’Hern, R., & Prouse, A. (1998) · PubMed · Cited by 34 · see citing works
Pundzius, J., Skaudickas, D., Boguševičius, A., et al. (2002) · The European Journal of Surgery · doi:10.1080/11024150252884287 · Cited by 32 · see citing works
Jousimaa, J., Mākelä, M., Kunnamo, I., et al. (2002) · PubMed · Cited by 31 · see citing works
Hogg, W. E., Bass, M. J., Calonge, N., Crouch, H., & Satenstein, G. (1998) · PubMed Central · Cited by 31 · see citing works
Frances, C. D. (2001) · Western Journal of Medicine · doi:10.1136/ewjm.175.3.165 · Cited by 31 · see citing works
An interactive voice response system to enhance antidepressant medication compliance.
Stuart, G. W., Laraia, M. T., Ornstein, S. M., & Nietert, P. J. (2003) · PubMed · Cited by 28 · see citing works
Schrezenmeir, J., Dirting, K., & Papazov, P. (2002) · Computer Methods and Programs in Biomedicine · doi:10.1016/s0169-2607(02)00034-2 · Cited by 27 · see citing works
Boukhors, Y., RABASA-LHORET, R. P. R., Langelier, H., Soultan, M., Lacroix, A., & Chiasson, J. L. (2003) · Diabetes & Metabolism · doi:10.1016/s1262-3636(07)70078-2 · Cited by 25 · see citing works
Randomized study of online vaccine reminders in adult primary care.
Flanagan, J. R., Doebbeling, B. N., Dawson, J. D., & Beekmann, S. E. (1999) · PubMed Central · Cited by 22 · see citing works
A framework for classifying decision support systems.
Sim, I., & Berlin, A. (2003) · PubMed · Cited by 21 · see citing works
Self-citationA controlled trial of three referral methods for patients with third molars.
Goodey, R. D., Brickley, M. R., Hill, C. M., et al. (2000) · BDJ · doi:10.1038/sj.bdj.4800828 · Cited by 16 · see citing works
Bennett, J. W., Glasziou, P., Mar, C. D., et al. (2003) · PubMed · Cited by 13 · see citing works
A computerized decision support system for ovarian stimulation by gonadotropins
Lesourd, F., Avril, C., Boujennah, A., & Parinaud, J. (2002) · Fertility and Sterility · doi:10.1016/s0015-0282(01)03231-9 · Cited by 13 · see citing works
Characteristics of Outpatient Clinical Decision Support Systems: A Taxonomic Description
Berlin, A., Sorani, M. D., & Ida, S. (2004) · Studies in health technology and informatics · doi:10.3233/978-1-60750-949-3-578 · Cited by 10 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 7
- Kuperman G, Sittig D, Shabot M. Clinical decision support for hospital and critical care. J Healthc Inf Manage 1999;13(2):81-96.
- Broverman CA. Standards for clinical decision support systems. J Healthc Inf Manage 1999;13(2):23-31.
- Perreault L. A pragmatic framework for understanding decision support. J Healthc Inf Manage 1999;13(2):5-21.
- Teich JM. Inpatient order management. J Healthc Inf Manage 1999;13(2):97-110.
- Burack RC, Gimotty PA, George J, McBride S, Moncrease A, Simon MS, et al. How reminders given to patients and physicians affected pap smear use in a health maintenance organization: results of a randomized controlled trial. Cancer 1998;82(12):2391-400.
- Shojania K, Yokoe D, Platt R, Fiskio J, Ma'luf N, Bates D. Reducing vancomycin use utilizing a computer guidline. J Am Med Inform Assoc 1998;5:554-62.
- Williams RB, Boles M, Johnson R. A Patient-Initiated System for Preventive Health Care. Arch Fam Med 1998;7:338-45.
Cited by
Sorted by citations
The Technology Acceptance Model: Its past and its future in health care
Holden, R. J., & Karsh, B.-T. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.07.002 · Cited by 2,669 · see citing works
The Impact of eHealth on the Quality and Safety of Health Care: A Systematic Overview
Black, A. D., Car, J., Pagliari, C., Anandan, C., Cresswell, K., Bokun, T., McKinstry, B., Procter, R., Majeed, A., & Sheikh, A. (2011) · PLoS Medicine · doi:10.1371/journal.pmed.1000387 · Cited by 1,442 · see citing works
Free, C., Phillips, G., Felix, L., Galli, L., Patel, V., & Edwards, P. (2010) · BMC Research Notes · doi:10.1186/1756-0500-3-250 · Cited by 539 · see citing works
“PDAs for data collection in healthcare and health research [1,8-10] and to support medical education and clinical practice [2,11-13].”
Improving tractability of Clinical Decision Support system
Tan, T. Z., Ng, G. S., & Quek, C. (2008) · 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence) · doi:10.1109/ijcnn.2008.4634072 · Cited by 3 · see citing works
- Authors
- Berlin, A., Sorani, M., & Sim, I.
- Affiliations
- University of California, San Francisco (US)
- Corresponding author
- Sim, Ida
- ORCID
- Sim, Ida
- Venue
- Journal of Biomedical Informatics
- Publisher
- Elsevier BV
- Published
- 2006-12
- Volume
- 39
- Issue
- 6
- Pages
- 656-667
- ISSN
- 1532-0464 (ISSN-L)
- OpenAlex
- W2159312907
- Cited by
- 123
- References
- 71
- Citations by year
- 2024: 3 · 2023: 4 · 2022: 3 · 2021: 3 · 2020: 7 · 2019: 9 · 2018: 6 · 2017: 6 · 2016: 11 · 2015: 10 · 2014: 3 · 2013: 12 · 2012: 8
- FWCI
- 13.86
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- Locations
- Journal of Biomedical Informatics (publishedVersion)
- PubMed (publishedVersion)
- eScholarship (California Digital Library) (submittedVersion)
- Funders
- National Institute of Mental Health (MH 060482, R25MH060482)
- U.S. National Library of Medicine (LM 06780, R01LM006780)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Technology and Human Factors in Education and Health
- Keywords
- clinical decision support systems, information delivery, generalizability of trial results
- MeSH terms
- Electronic Data Processing, Decision Making, Computer-Assisted, Humans, Terminology as Topic, Outcome and Process Assessment, Health Care, Preventive Medicine, Randomized Controlled Trials as Topic, Databases, Bibliographic, Medical Records Systems, Computerized, Decision Support Systems, Clinical, Internet, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
