The DOI matches its Crossref record; 82 of the 82 references it cites were found in a registry or an index.
Instructional Design Variations in Internet-Based Learning for Health Professions Education: A Systematic Review and Meta-Analysis
Abstract
PURPOSE: A recent systematic review (2008) described the effectiveness of Internet-based learning (IBL) in health professions education. A comprehensive synthesis of research investigating how to improve IBL is needed. This systematic review sought to provide such a synthesis. METHOD: The authors searched MEDLINE, CINAHL, EMBASE, Web of Science, Scopus, ERIC, TimeLit, and the University of Toronto Research and Development Resource Base for articles published from 1990 through November 2008. They included all studies quantifying the effect of IBL compared with another Internet-based or computer-assisted instructional intervention on practicing and student physicians, nurses, pharmacists, dentists, and other health professionals. Reviewers working independently and in duplicate abstracted information, coded study quality, and grouped studies according to inductively identified themes. RESULTS: From 2,705 articles, the authors identified 51 eligible studies, including 30 randomized trials. The pooled effect size (ES) for learning outcomes in 15 studies investigating high versus low interactivity was 0.27 (95% confidence interval, 0.08-0.46; P = .006). Also associated with higher learning were practice exercises (ES 0.40 [0.08-0.71; P = .01]; 10 studies), feedback (ES 0.68 [0.01-1.35; P = .047]; 2 studies), and repetition of study material (ES 0.19 [0.09-0.30; P < .001]; 2 studies). The ES was 0.26 (-0.62 to 1.13; P = .57) for three studies examining online discussion. Inconsistency was large (I(2) >or=89%) in most analyses. Meta-analyses for other themes generally yielded imprecise results. CONCLUSIONS: Interactivity, practice exercises, repetition, and feedback seem to be associated with improved learning outcomes, although inconsistency across studies tempers conclusions. Evidence for other instructional variations remains inconclusive.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical power analysis for the behavioral sciences
(1990) · Computers, Environment and Urban Systems · doi:10.1016/0198-9715(90)90050-4 · Cited by 65,269
Measuring inconsistency in meta-analyses
Higgins, J. P. T., Thompson, S. G., Deeks, J. J., & Altman, D. G. (2003) · BMJ · doi:10.1136/bmj.327.7414.557 · Cited by 63,139
Statistical Power Analysis for the Behavioral Sciences (2nd ed.).
Lachenbruch, P. A., & Cohen, J. (1989) · Journal of the American Statistical Association · doi:10.2307/2290095 · Cited by 37,395
Intraclass correlations: Uses in assessing rater reliability.
Shrout, P. E., & Fleiss, J. L. (1979) · Psychological Bulletin · doi:10.1037/0033-2909.86.2.420 · Cited by 24,532
Meta-analysis of Observational Studies in Epidemiology<SUBTITLE>A Proposal for Reporting</SUBTITLE>
Stroup, D. F. (2000) · JAMA · doi:10.1001/jama.283.15.2008 · Cited by 20,623
(2014) · Medical Entomology and Zoology · Cited by 17,633
Methods of Meta-Analysis: Correcting Error and Bias in Research Findings
Schmidt, F. L., & Hunter, J. E. (2015) · doi:10.4135/9781483398105 · Cited by 5,542
Moher, D., Cook, D. J., Eastwood, S., Olkin, I., Rennie, D., & Stroup, D. F. (1999) · The Lancet · doi:10.1016/s0140-6736(99)04149-5 · Cited by 4,826
Self-citationThe Handbook of Research Synthesis.
Howel, D., Cooper, H., & Hedges, L. V. (1995) · Biometrics · doi:10.2307/2533295 · Cited by 3,642
Reconsidering Research on Learning from Media
Clark, R. E. (1983) · Review of Educational Research · doi:10.3102/00346543053004445 · Cited by 2,715
Morris, S. B., & DeShon, R. P. (2002) · Psychological Methods · doi:10.1037/1082-989x.7.1.105 · Cited by 2,502
Clark, R. C., Mayer, R. E., & Thalheimer, W. (2003) · Performance Improvement · doi:10.1002/pfi.4930420510 · Cited by 2,257
The Impact of E-Learning in Medical Education
Ruiz, J. G., Mintzer, M. J., & Leipzig, R. M. (2006) · Academic Medicine · doi:10.1097/00001888-200603000-00002 · Cited by 2,240
Parametric measures of effect size.
Rosenthal, R. (1994) · Cited by 1,847
The case of the misleading funnel plot
Lau, J., Ioannidis, J. P. A., Terrin, N., Schmid, C. H., & Olkin, I. (2006) · BMJ · doi:10.1136/bmj.333.7568.597 · Cited by 1,766
Meta-analysis of experiments with matched groups or repeated measures designs
Dunlap, W. P., Cortina, J. M., Vaslow, J. B., & Burke, M. J. (1996) · Psychological Methods · doi:10.1037/1082-989x.1.2.170 · Cited by 1,693
Internet-Based Learning in the Health Professions: A Meta-analysis
Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M. (2008) · JAMA · doi:10.1001/jama.300.10.1181 · Cited by 1,498
Self-citationTeaching Courses Online: A Review of the Research
Tallent-Runnels, M. K., Thomas, J. A., Lan, W. Y., Cooper, S., Ahern, T. C., Shaw, S. M., & Liu, X. (2006) · Review of Educational Research · doi:10.3102/00346543076001093 · Cited by 1,267
Kirkpatrick, D. L. (1996) · Training and development journal · Cited by 626
Dillon, A., & Gabbard, R. (1998) · Review of Educational Research · doi:10.2307/1170600 · Cited by 581
Web-based Learning: Sound Educational Method or Hype? A Review of the Evaluation Literature
CHUMLEY-JONES, H. S., DOBBIE, A., & ALFORD, C. L. (2002) · Academic Medicine · doi:10.1097/00001888-200210001-00028 · Cited by 508
Nicholson, D. T., Chalk, C., Funnell, W. R. J., & Daniel, S. J. (2006) · Medical Education · doi:10.1111/j.1365-2929.2006.02611.x · Cited by 473
Web-based learning: pros, cons and controversies
Cook, D. A. (2007) · Clinical Medicine · doi:10.7861/clinmedicine.7-1-37 · Cited by 437
Self-citationComputer assisted learning in undergraduate medical education
Greenhalgh, T. (2001) · BMJ · doi:10.1136/bmj.322.7277.40 · Cited by 423
eLearning: A review of Internet-based continuing medical education
Wutoh, R., Boren, S. A., & Balas, A. E. (2004) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.1340240105 · Cited by 409
A review of evaluation outcomes of web‐based continuing medical education
Curran, V. R., & Fleet, L. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02173.x · Cited by 317
The Research We Still Are Not Doing: An Agenda for the Study of Computer-Based Learning
Cook, D. A. (2005) · Academic Medicine · doi:10.1097/00001888-200506000-00005 · Cited by 289
Self-citationThe Efficacy of Computer Assisted Instruction (CAI): A Meta-Analysis
Fletcher-Flinn, C. M., & Gravatt, B. (1995) · Journal of Educational Computing Research · doi:10.2190/51d4-f6l3-jqhu-9m31 · Cited by 283
Kerfoot, B. P., DeWolf, W. C., Masser, B. A., Church, P. A., & Federman, D. D. (2007) · Medical Education · doi:10.1111/j.1365-2929.2006.02644.x · Cited by 278
Meta‐analysis combining parallel and cross‐over clinical trials. I: Continuous outcomes
Curtin, F., Altman, D. G., & Elbourne, D. (2002) · Statistics in Medicine · doi:10.1002/sim.1205 · Cited by 207
A Call for Outcomes Research in Medical Education
Chen, F. M., Bauchner, H., & Burstin, H. (2004) · Academic Medicine · doi:10.1097/00001888-200410000-00010 · Cited by 205
The failure of e-learning research to inform educational practice, and what we can do about it
Cook, D. A. (2009) · Medical Teacher · doi:10.1080/01421590802691393 · Cited by 201
Self-citationCampbell, M., Gibson, W., Hall, A., Richards, D., & Callery, P. (2008) · International Journal of Nursing Studies · doi:10.1016/j.ijnurstu.2006.12.011 · Cited by 168
The Effectiveness of Computer-Based Adult Education: A Meta-Analysis
Kulik, C. C., Kulik, J. A., & Shwalb, B. J. (1986) · Journal of Educational Computing Research · doi:10.2190/02hm-xcwg-q1vy-5emq · Cited by 156
Kopp, V., Stark, R., & Fischer, M. R. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03122.x · Cited by 147
Prystowsky, J. B., & Bordage, G. (2001) · Medical Education · doi:10.1046/j.1365-2923.2001.00910.x · Cited by 147
Bearman, M., Cesnik, B., & Liddell, M. J. (2001) · Medical Education · doi:10.1046/j.1365-2923.2001.00999.x · Cited by 131
The research we should be doing
Friedman, C. P. (1994) · Academic Medicine · doi:10.1097/00001888-199406000-00005 · Cited by 126
Do medical students watch video clips in eLearning and do these facilitate learning?
Romanov, K., & Nevgi, A. (2007) · Medical Teacher · doi:10.1080/01421590701542119 · Cited by 120
Where are we with Web-based learning in medical education?
Cook, D. A. (2006) · Medical Teacher · doi:10.1080/01421590601028854 · Cited by 118
Self-citationExperiences with different integration strategies of case-based e-learning
Hege, I., Ropp, V., Adler, M., Radon, K., Mäsch, G., Lyon, H. C., & Fischer, M. R. (2007) · Medical Teacher · doi:10.1080/01421590701589193 · Cited by 108
Cook, D. A., Thompson, W. G., Thomas, K. G., Thomas, M. R., & Pankratz, V. S. (2006) · Academic Medicine · doi:10.1097/00001888-200603000-00005 · Cited by 105
Self-citationMaag, M. M. (2004) · CIN Computers Informatics Nursing · doi:10.1097/00024665-200401000-00007 · Cited by 99
Quantifying the Literature of Computer-aided Instruction in Medical Education
Adler, M., & Johnson, K. B. (2000) · Academic Medicine · doi:10.1097/00001888-200010000-00021 · Cited by 89
A review of evaluative studies of computer-based learning in nursing education
Lewis, M. J., Davies, R., Jenkins, D., & Tait, M. I. (2001) · Nurse Education Today · doi:10.1054/nedt.2000.0494 · Cited by 89
Computer-Based Instruction and Health Professions Education: A Meta-Analysis of Outcomes
Cohen, P. A., & Dacanay, L. S. (1992) · Evaluation & the Health Professions · doi:10.1177/016327879201500301 · Cited by 87
The inadequacy of recent research on computer-assisted instruction
Keane, D. R., Norman, G. R., & Vickers, J. (1991) · Academic Medicine · doi:10.1097/00001888-199108000-00005 · Cited by 86
The Effect of Communication on Nursing Student Outcomes in a Web-Based Course
Frith, K. H., & Kee, C. C. (2003) · Journal of Nursing Education · doi:10.3928/0148-4834-20030801-06 · Cited by 84
Kerfoot, B. P., Baker, H. E., Koch, M. O., Connelly, D., Joseph, D. B., & Ritchey, M. L. (2007) · The Journal of Urology · doi:10.1016/j.juro.2006.11.074 · Cited by 83
Chan, D., Leclair, K., & Kaczorowski, J. A. (1999) · PubMed · Cited by 79
Online spaced education to teach urology to medical students: a multi-institutional randomized trial
Kerfoot, B. P., & Brotschi, E. A. (2008) · The American Journal of Surgery · doi:10.1016/j.amjsurg.2007.10.026 · Cited by 79
Cook, D. A., Thompson, W. G., Thomas, K. G., & Thomas, M. R. (2007) · Advances in Health Sciences Education · doi:10.1007/s10459-007-9089-8 · Cited by 76
Self-citationCook, D. A., Gelula, M. H., Dupras, D. M., & Schwartz, A. (2007) · Medical Education · doi:10.1111/j.1365-2923.2007.02822.x · Cited by 76
Self-citationSurgical e‐learning: validation of multimedia web‐based lectures
Ridgway, P. F., Sheikh, A., Sweeney, K., Evoy, D., McDermott, E. W. M., Felle, P., Hill, A. D. K., & O’Higgins, N. J. (2007) · Medical Education · doi:10.1111/j.1365-2929.2006.02669.x · Cited by 69
Moridani, M. Y. (2007) · American Journal of Pharmaceutical Education · doi:10.5688/aj710116 · Cited by 66
Multicomponent Internet continuing medical education to promote chlamydia screening
Allison, J. J., Kiefe, C. I., Wall, T. C., Casebeer, L. L., Ray, M. N., Spettell, C. M., Hook, E. W., Oh, M. K., Person, S. D., & Weissman, N. (2005) · American Journal of Preventive Medicine · doi:10.1016/j.amepre.2004.12.013 · Cited by 65
Adapting Web-based Instruction to Residents’ Knowledge Improves Learning Efficiency
Cook, D. A., Beckman, T. J., Thomas, K. G., & Thompson, W. G. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0541-0 · Cited by 57
Self-citationMeta‐analysis of quasi‐experimental research: are systematic narrative reviews indicated?
Colliver, J. A., Kucera, K., & Verhulst, S. J. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03144.x · Cited by 49
Using Synchronous Software in Web-based Nursing Courses
Little, B. B., Passmore, D., & Schullo, S. (2006) · CIN Computers Informatics Nursing · doi:10.1097/00024665-200611000-00005 · Cited by 45
Duffy, T., Gilbert, I., Kennedy, D. L., & Kwong, P. W. (2002) · Research in Learning Technology · doi:10.3402/rlt.v10i1.11304 · Cited by 40
Romanov, K., & Nevgi, A. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.06.004 · Cited by 40
Mukohara, K., & Schwartz, M. D. (2005) · Medical Education · doi:10.1111/j.1365-2929.2005.02109.x · Cited by 39
Kemper, K. J., Gardiner, P. M., Gobble, J., Mitra, A., & Woods, C. R. (2006) · BMC Medical Education · doi:10.1186/1472-6920-6-2 · Cited by 38
Computer-assisted learning in critical care: From ENIAC to HAL
Tegtmeyer, K., Ibsen, L. M., & Goldstein, B. (2001) · Critical Care Medicine · doi:10.1097/00003246-200108001-00006 · Cited by 38
Bednar, E. D., Hannum, W. M., Firestone, A. R., Silveira, A. M., Cox, T. D., & Proffit, W. R. (2007) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2007.06.008 · Cited by 38
Kerfoot, B. P. (2008) · The Journal of Urology · doi:10.1016/j.juro.2008.01.126 · Cited by 36
The effects of an interactive software application on the self‐assessment ability of dental students
Mattheos, N., Nattestad, A., CHRISTERSSON, C. E., Jansson, H., & Attström, R. (2004) · European Journal Of Dental Education · doi:10.1111/j.1600-0579.2004.00325.x · Cited by 33
Mind the gap: some reasons why medical education research is different from health services research
Shea, J. A. (2001) · Medical Education · doi:10.1046/j.1365-2923.2001.00913.x · Cited by 33
Learning & retention in adaptive serious games.
Bergeron, B. P. (2008) · PubMed · Cited by 33
Methods to improve teaching interdisciplinary teamwork through computer conferencing.
Becker, E. A., & Godwin, E. M. (2005) · PubMed · Cited by 31
Proposed Standards for Medical Education Submissions to the Journal of General Internal Medicine
Cook, D. A., Bowen, J. L., Gerrity, M., Kalet, A. L., Kogan, J. R., Spickard, A., & Wayne, D. B. (2008) · Journal of General Internal Medicine · doi:10.1007/s11606-008-0676-z · Cited by 30
Self-citationThe effects of immediate online feedback upon diagnostic performance
Papa, F. J., Aldrich, D. G., & Schumacker, R. E. (1999) · Academic Medicine · doi:10.1097/00001888-199910000-00027 · Cited by 28
Comparative Evaluation of Multimedia Driven, Interactive, and Case-Based Teaching in Heart Surgery
Friedl, R., Höppler, H., Ecard, K., Scholz, W., Hannekum, A., Oechsner, W., & Stracke, S. (2006) · The Annals of Thoracic Surgery · doi:10.1016/j.athoracsur.2006.05.118 · Cited by 27
Use of recorded interactive seminars in orthodontic distance education
Miller, K. T., Hannum, W. M., Morley, T., & Proffit, W. R. (2007) · American Journal of Orthodontics and Dentofacial Orthopedics · doi:10.1016/j.ajodo.2007.03.015 · Cited by 25
Teaching Brief Interventions for Smoking Cessation via an Interactive Computer-based Tutorial
Carpenter, K., Watson, J. M., Raffety, B. D., & Chabal, C. (2003) · Journal of Health Psychology · doi:10.1177/1359105303008001450 · Cited by 21
Change management in primary care: design and evaluation of an internet‐delivered course
Fox, N. J., O’Rourke, A., Roberts, C., & Walker, J. (2001) · Medical Education · doi:10.1046/j.1365-2923.2001.00974.x · Cited by 20
Introducing resident doctors to complexity in ambulatory medicine
Cook, D. A., Beckman, T. J., Thomas, K. G., & Thompson, W. G. (2008) · Medical Education · doi:10.1111/j.1365-2923.2008.03108.x · Cited by 19
Self-citationTunuguntla, R., Rodríguez, O., Ruiz, J. G., Qadri, S. S., Mintzer, M. J., van Zuilen, M. H., & Roos, B. A. (2008) · Medical Teacher · doi:10.1080/01421590802263508 · Cited by 19
Duffy, T., Gilbert, I., Kennedy, D. L., & Kwong, P. W. (2002) · ALT-J · doi:10.1080/0968776020100110 · Cited by 17
Evaluation of hypertext in an activity learning environment
Scherly, D., Roux, L., & Dillenbourg, P. (2000) · Journal of Computer Assisted Learning · doi:10.1046/j.1365-2729.2000.00124.x · Cited by 9
Evaluating the serial migration of an existing required course to the World Wide Web
Schaad, D. C., Walker, E. A., Wolf, F. M., Brock, D. M., Thielke, S. M., & Oberg, L. (1999) · Academic Medicine · doi:10.1097/00001888-199910000-00048 · Cited by 8
Evaluation of an Online Analogical Patient Simulation Program
Thompson, G. A., Morrison, R., Holyoak, K. J., & Clark, T. K. (2006) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2006.85 · Cited by 6
Research works printed but not linked: 17
- Kirkpatrick, 1996, Training Dev, 50, 54
- Kaplan, 1996, Am J Pharm Educ, 60, 422, 10.1016/S0002-9459(24)04687-4 · doi:10.1016/s0002-9459(24)04687-4
- Chan, 1999, MD Comput, 16, 54
- Jedlicka, 2002, J Allied Health, 31, 247
- Chao, 2003, J Telemed Telecare, 9, S4, 10.1258/135763303322196141 · doi:10.1258/135763303322196141
- Lemaire, 2003, J Telemed Telecare, 9, 17, 10.1258/135763303321159648 · doi:10.1258/135763303321159648
- Spickard, 2004, Med Educ, 38, 787, 10.1111/j.1365-2929.2004.01824.x · doi:10.1111/j.1365-2929.2004.01824.x
- Virvou, 2004, 1, 48
- Becker, 2005, J Allied Health, 34, 169
- Brunetaud, 2005, Int J Med Inform, 74, 209, 10.1016/j.ijmedinf.2004.06.006 · DOI
- Schittek Janda, 2005, Eur J Dent Educ, 9, 53, 10.1111/j.1600-0579.2004.00366.x · DOI
- Blackmore, 2006, Int J Psychother, 10, 35
- Kerfoot, 2006, Med Educ, 40, 1137, 10.1111/j.1365-2929.2006.02588.x · doi:10.1111/j.1365-2929.2006.02588.x
- Al-Rawi, 2007, Dentomaxillofac Radiol, 36, 459, 10.1259/dmfr/25560514 · doi:10.1259/dmfr/25560514
- Bergeron, 2008, Stud Health Technol Inform, 132, 26
- Walker, 2008, Educ Gerontol, 34, 691, 10.1080/03601270802055778 · doi:10.1080/03601270802055778
- Dillon, 1998, Rev Educ Res, 68, 322, 10.3102/00346543068003322 · doi:10.3102/00346543068003322 · Paper page
Cited by
Cook, D. A., & Reed, D. A. (2015) · Academic Medicine · doi:10.1097/acm.0000000000000786 · Cited by 781
Self-citationRegmi, K., & Jones, L. (2020) · BMC Medical Education · doi:10.1186/s12909-020-02007-6 · Cited by 751
Cook, D. A., Hamstra, S. J., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J., & Hatala, R. (2013) · Medical Teacher · doi:10.3109/0142159x.2012.714886 · Cited by 709
Self-citationBaticulon, R. E., Sy, J. J., Alberto, N. R. I., Baron, M. B. C., Mabulay, R. E. C., Rizada, L. G. T., Tiu, C. J. S., Clarion, C. A., & Reyes, J. C. B. (2021) · Medical Science Educator · doi:10.1007/s40670-021-01231-z · Cited by 706
The Effectiveness of Blended Learning in Health Professions: Systematic Review and Meta-Analysis
Liu, Q., Peng, W., Zhang, F., Hu, R., Li, Y., & Yan, W. (2016) · Journal of Medical Internet Research · doi:10.2196/jmir.4807 · Cited by 691
Metadata
- Authors
- Cook, D. A., Levinson, A. J., Garside, S., Dupras, D. M., Erwin, P. J., & Montori, V. M.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2010-05
- Volume
- 85
- Issue
- 5
- Pages
- 909-922
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W2085403719
- Cited by
- 547
- References
- 82
- FWCI
- 4.00
- Type
- journal article
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.