Evidence appraisal: a scoping review, conceptual framework, and research agenda
Open accessThe DOI matches its Crossref record; 194 of the 197 references it cites were found in a registry or an index.
Abstract
Objective: Critical appraisal of clinical evidence promises to help prevent, detect, and address flaws related to study importance, ethics, validity, applicability, and reporting. These research issues are of growing concern. The purpose of this scoping review is to survey the current literature on evidence appraisal to develop a conceptual framework and an informatics research agenda. Methods: We conducted an iterative literature search of Medline for discussion or research on the critical appraisal of clinical evidence. After title and abstract review, 121 articles were included in the analysis. We performed qualitative thematic analysis to describe the evidence appraisal architecture and its issues and opportunities. From this analysis, we derived a conceptual framework and an informatics research agenda. Results: We identified 68 themes in 10 categories. This analysis revealed that the practice of evidence appraisal is quite common but is rarely subjected to documentation, organization, validation, integration, or uptake. This is related to underdeveloped tools, scant incentives, and insufficient acquisition of appraisal data and transformation of the data into usable knowledge. Discussion: The gaps in acquiring appraisal data, transforming the data into actionable information and knowledge, and ensuring its dissemination and adoption can be addressed with proven informatics approaches. Conclusions: Evidence appraisal faces several challenges, but implementing an informatics research agenda would likely help realize the potential of evidence appraisal for improving the rigor and value of clinical evidence.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“As vast evidence bases such as PubMed grows exponentially and rapidly, evidence retrieval and appraisal become extremely difficult due to information overload.3”
From A neuro-symbolic method for understanding free-text medical evidence. (2021) · cites this paper as [3]
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000097 · Cited by 64,042
Preferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742
The Cochrane Collaboration's tool for assessing risk of bias in randomised trials
Higgins, J. P. T., Altman, D. G., Gotzsche, P. C., Juni, P., Moher, D., Oxman, A. D., Savovic, J., Schulz, K. F., Weeks, L., Sterne, J. A. C., et al. (2011) · BMJ · doi:10.1136/bmj.d5928 · Cited by 33,817
Dermyshi, E., Wang, Y., Yan, C., Hong, W., Qiu, G., Gong, X., & Zhang, T. (2017) · Neonatology · doi:10.1159/000454668 · Cited by 24,451
Wang, G., Ma, R., Qiao, G., Wada, K., Aizawa, Y., & Satoh, T. (2015) · Integrative Medicine International · doi:10.1159/000368408 · Cited by 18,657
MIMIC-III, a freely accessible critical care database
Johnson, A. E. W., Pollard, T. J., Shen, L., Lehman, L.-W. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., & Mark, R. G. (2016) · Scientific Data · doi:10.1038/sdata.2016.35 · Cited by 8,141
GRADE guidelines: 4. Rating the quality of evidence—study limitations (risk of bias)
Guyatt, G. H., Oxman, A. D., Vist, G., Kunz, R., Brozek, J., Alonso-Coello, P., Montori, V., Akl, E. A., Djulbegovic, B., & Falck-Ytter, Y. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.07.017 · Cited by 2,798
Increasing value and reducing waste in research design, conduct, and analysis
Ioannidis, J. P. A., Greenland, S., Hlatky, M. A., Khoury, M. J., Macleod, M. R., Moher, D., Schulz, K. F., & Tibshirani, R. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62227-8 · Cited by 1,597
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542
How to increase value and reduce waste when research priorities are set
Chalmers, I., Bracken, M. B., Djulbegovic, B., Garattini, S., Grant, J., Gülmezoglu, A. M., Howells, D. W., Ioannidis, J. P. A., & Oliver, S. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62229-1 · Cited by 1,436
Reducing waste from incomplete or unusable reports of biomedical research
Glasziou, P., Altman, D. G., Bossuyt, P., Boutron, I., Clarke, M., Julious, S., Michie, S., Moher, D., & Wager, E. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62228-x · Cited by 1,319
Policy: NIH plans to enhance reproducibility
Collins, F. S., & Tabak, L. A. (2014) · Nature · doi:10.1038/505612a · Cited by 1,247
A call for transparent reporting to optimize the predictive value of preclinical research
Landis, S. C., Amara, S. G., Asadullah, K., Austin, C. P., Blumenstein, R., Bradley, E. W., Crystal, R. G., Darnell, R. B., Ferrante, R. J., Fillit, H., et al. (2012) · Nature · doi:10.1038/nature11556 · Cited by 1,196
Savović, J., Jones, H. E., Altman, D. G., Harris, R. J., Jüni, P., Pildal, J., Als-Nielsen, B., Balk, E. M., Gluud, C., Gluud, L. L., et al. (2012) · Annals of Internal Medicine · doi:10.7326/0003-4819-157-6-201209180-00537 · Cited by 1,049
Criteria for evaluating evidence on public health interventions
Rychetnik, L., Frommer, M., Hawe, P., & Shiell, A. (2002) · Journal of Epidemiology and Community Health · doi:10.1136/jech.56.2.119 · Cited by 1,042
Biomedical research: increasing value, reducing waste.
Macleod, M. R., Michie, S., Roberts, I., Dirnagl, U., Chalmers, I., Ioannidis, J. P. A., Salman, R. A.-S., Chan, A.-W., & Glasziou, P. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62329-6 · Cited by 1,003
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.1993.03510210084032 · Cited by 944
The scandal of poor medical research.
Altman, D. G. (1994) · BMJ · doi:10.1136/bmj.308.6924.283 · Cited by 897
Increasing value and reducing waste: addressing inaccessible research
Chan, A.-W., Song, F., Vickers, A., Jefferson, T., Dickersin, K., Gøtzsche, P. C., Krumholz, H. M., Ghersi, D., & van der Worp, H. B. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62296-5 · Cited by 887
The ClinicalTrials.gov Results Database — Update and Key Issues
Zarin, D. A., Tse, T., Williams, R. J., Califf, R. M., & Ide, N. C. (2011) · New England Journal of Medicine · doi:10.1056/nejmsa1012065 · Cited by 838
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.270.17.2093 · Cited by 655
The Learning Healthcare System: Workshop Summary
Olsen, L., Aisner, D. L., & McGinnis, J. M. (2013) · Cited by 582
The Continuing Unethical Conduct of Underpowered Clinical Trials
Halpern, S. D. (2002) · JAMA · doi:10.1001/jama.288.3.358 · Cited by 547
Achieving a Nationwide Learning Health System
Friedman, C. P., Wong, A. K., & Blumenthal, D. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001456 · Cited by 539
Increasing value and reducing waste in biomedical research regulation and management
Salman, R. A.-S., Beller, E., Kagan, J., Hemminki, E., Phillips, R. S., Savulescu, J., Macleod, M., Wisely, J., & Chalmers, I. (2014) · The Lancet · doi:10.1016/s0140-6736(13)62297-7 · Cited by 534
Meta-research: Evaluation and Improvement of Research Methods and Practices
Ioannidis, J. P. A., Fanelli, D., Dunne, D. D., & Goodman, S. N. (2015) · PLOS Biology · doi:10.1371/journal.pbio.1002264 · Cited by 495
Ferreira-González, I., Permanyer-Miralda, G., Domingo-Salvany, A., Busse, J. W., Heels-Ansdell, D., Montori, V. M., Akl, E. A., Bryant, D. M., Alonso-Coello, P., Alonso, J., et al. (2007) · BMJ · doi:10.1136/bmj.39136.682083.ae · Cited by 473
Users' Guides to the Medical Literature: I. How to Get Started
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.1993.03510170083036 · Cited by 469
van Rooyen, S., Godlee, F., Evans, S., Black, N., & Smith, R. (1999) · BMJ · doi:10.1136/bmj.318.7175.23 · Cited by 415
Scientific literature: Information overload
Landhuis, E. (2016) · Nature · doi:10.1038/nj7612-457a · Cited by 324
The journal club and medical education: over one hundred years of unrecorded history
Linzer, M. (1987) · Postgraduate Medical Journal · doi:10.1136/pgmj.63.740.475 · Cited by 321
How to run an effective journal club: a systematic review
Deenadayalan, Y., Grimmer‐Somers, K., Prior, M., & Kumar, S. (2008) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2008.01050.x · Cited by 308
Journals unite for reproducibility
McNutt, M. (2014) · Science · doi:10.1126/science.aaa1724 · Cited by 300
Eyre-Walker, A., & Stoletzki, N. (2013) · PLoS Biology · doi:10.1371/journal.pbio.1001675 · Cited by 297
What Are the Effects of Teaching Evidence-Based Health Care (EBHC)? Overview of Systematic Reviews
Young, T., Rohwer, A., Volmink, J., & Clarke, M. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086706 · Cited by 254
Sharing Clinical Trial Data: A Proposal From the International Committee of Medical Journal Editors
Taichman, D. B., Backus, J., Baethge, C., Bauchner, H., de Leeuw, P. W., Drazen, J. M., Fletcher, J., Frizelle, F. A., Groves, T., Haileamlak, A., et al. (2016) · JAMA · doi:10.1001/jama.2015.18164 · Cited by 254
Getting the methods right--the foundation of patient-centered outcomes research
Gabriel, S. E., & Normand, S.-L. T. (2012) · New England Journal of Medicine · doi:10.1056/nejmp1207437 · Cited by 252
Comparison of protocols and registry entries to published reports for randomised controlled trials
Dwan, K., Altman, D. G., Cresswell, L., Blundell, M., Gamble, C. L., & Williamson, P. R. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000031.pub2 · Cited by 243
Publication Pressure and Scientific Misconduct in Medical Scientists
Tijdink, J. K., Verbeke, R., & Smulders, Y. M. (2014) · Journal of Empirical Research on Human Research Ethics · doi:10.1177/1556264614552421 · Cited by 221
A review of journal clubs in postgraduate medical education
Alguire, P. C. (1998) · Journal of General Internal Medicine · doi:10.1046/j.1525-1497.1998.00102.x · Cited by 207
Washington, A. E., & Lipstein, S. H. (2011) · New England Journal of Medicine · doi:10.1056/nejmp1109407 · Cited by 199
Patient-Centered Outcomes Research Institute: The Intersection of Science and Health Care
Clancy, C., & Collins, F. S. (2010) · Science Translational Medicine · doi:10.1126/scitranslmed.3001235 · Cited by 190
Reanalyses of Randomized Clinical Trial Data
Ebrahim, S., Sohani, Z. N., Montoya, L., Agarwal, A., Thorlund, K., Mills, E. J., & Ioannidis, J. P. A. (2014) · JAMA · doi:10.1001/jama.2014.9646 · Cited by 184
Incongruence between test statistics and P values in medical papers
García-Berthou, E., & Alcaraz, C. (2004) · BMC Medical Research Methodology · doi:10.1186/1471-2288-4-13 · Cited by 181
The CTSA Program at NIH: Opportunities for Advancing Clinical and Translational Research
Leshner, A. I., Terry, S. F., Schultz, A. M., & Liverman, C. T. (2013) · University Libraries (University of Maryland) · doi:10.13016/7hfz-ttmw · Cited by 168
The journal club in postgraduate medical education: A systematic review
Ebbert, J. O., Montori, V. M., & Schultz, H. J. (2001) · Medical Teacher · doi:10.1080/01421590120075670 · Cited by 168
International Urology Journal Club via Twitter: 12-Month Experience
Thangasamy, I. A., Leveridge, M., Davies, B. J., Finelli, A., Stork, B., & Woo, H. H. (2014) · European Urology · doi:10.1016/j.eururo.2014.01.034 · Cited by 161
Crowe, M., Sheppard, L., & Campbell, A. (2011) · International Journal of Evidence-Based Healthcare · doi:10.1111/j.1744-1609.2011.00237.x · Cited by 160
Teixeira da Silva, J. A., & Dobránszki, J. (2015) · Accountability in Research · doi:10.1080/08989621.2014.899909 · Cited by 158
The Quality of Registration of Clinical Trials
Viergever, R. F., & Ghersi, D. (2011) · PLoS ONE · doi:10.1371/journal.pone.0014701 · Cited by 150
How Are Internal Medicine Residency Journal Clubs Organized, and What Makes Them Successful?
Sidorov, J. E. (1995) · Archives of Internal Medicine · doi:10.1001/archinte.1995.00430110111012 · Cited by 137
Gross, C. G. (2015) · Annual Review of Psychology · doi:10.1146/annurev-psych-122414-033437 · Cited by 136
The Evolution of Trial Registries and Their Use to Assess the Clinical Trial Enterprise
Dickersin, K. (2012) · JAMA · doi:10.1001/jama.2012.4230 · Cited by 127
Data fraud in clinical trials.
George, S. L., & Buyse, M. (2015) · Clinical Investigation · doi:10.4155/cli.14.116 · Cited by 126
Ahmadi, N., McKenzie, M. E., MacLean, A. R., Brown, C. J., Mastracci, T. M., & McLeod, R. S. (2011) · Journal of surgical education · doi:10.1016/j.jsurg.2011.07.004 · Cited by 125
Evidence‐based medicine training in graduate medical education: past, present and future
Green, M. L. (2000) · Journal of Evaluation in Clinical Practice · doi:10.1046/j.1365-2753.2000.00239.x · Cited by 118
Roberts, M. J., Perera, M., Lawrentschuk, N., Romanic, D., Papa, N., & Bolton, D. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4194 · Cited by 114
Clark, E., Burkett, K., & Stanko‐Lopp, D. (2009) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2009.01314.x · Cited by 114
Seymour, B., Kinn, S., & Sutherland, N. (2003) · Journal of Advanced Nursing · doi:10.1046/j.1365-2648.2003.02618.x · Cited by 110
Burchett, H., Umoquit, M., & Dobrow, M. (2011) · Journal of Health Services Research & Policy · doi:10.1258/jhsrp.2011.010124 · Cited by 110
When are ‘positive’ clinical trials in oncology truly positive?
Ocana, A., & Tannock, I. F. (2011) · JNCI Journal of the National Cancer Institute · doi:10.1093/jnci/djq463 · Cited by 110
Pöschl, U. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00033 · Cited by 105
Peer review for biomedical publications: we can improve the system
Stahel, P. F., & Moore, E. E. (2014) · BMC Medicine · doi:10.1186/s12916-014-0179-1 · Cited by 103
Designing, Conducting, and Evaluating Journal Clubs in Orthopaedic Surgery
Dirschl, D. R., Tornetta, P., & Bhandari, M. (2003) · Clinical Orthopaedics and Related Research · doi:10.1097/01.blo.0000081203.51121.25 · Cited by 98
A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal
Coomarasamy, A., Taylor, R. S., & Khan, K. S. (2003) · Medical Teacher · doi:10.1080/0142159021000061468 · Cited by 94
Research Blogging: Indexing and Registering the Change in Science 2.0
Fausto, S., Machado, F. A., Bento, L. F. J., Iamarino, A., Nahas, T. R., & Munger, D. S. (2012) · PLoS ONE · doi:10.1371/journal.pone.0050109 · Cited by 85
Post-Publication Peer Review: Opening Up Scientific Conversation
Hunter, J. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00063 · Cited by 84
Kriegeskorte, N. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00079 · Cited by 84
Bellomo, R., & Bagshaw, S. M. (2006) · Critical Care · doi:10.1186/cc5045 · Cited by 76
Ethical dilemmas in scientific publication: pitfalls and solutions for editors
Gollogly, L., & Momen, H. (2006) · Revista de Saúde Pública · doi:10.1590/s0034-89102006000400004 · Cited by 75
Reviewing post-publication peer review
Knoepfler, P. (2015) · Trends in Genetics · doi:10.1016/j.tig.2015.03.006 · Cited by 73
Ways of using evidence‐based medicine in general practice
Mar, C. D., & Glasziou, P. (2001) · The Medical Journal of Australia · doi:10.5694/j.1326-5377.2001.tb143313.x · Cited by 70
Complementary or alternative medicine in cancer care—myths and realities
Deng, G. E., & Cassileth, B. R. (2013) · Nature Reviews Clinical Oncology · doi:10.1038/nrclinonc.2013.125 · Cited by 70
Postpublication Criticism and the Shaping of Clinical Knowledge
Horton, R. C. (2002) · JAMA · doi:10.1001/jama.287.21.2843 · Cited by 68
Lachance, C. (2014) · The Journal of Continuing Education in Nursing · doi:10.3928/00220124-20141120-01 · Cited by 67
Postgraduate journal club as a means of promoting evidence-based obstetrics and gynaecology
Khan, K. S. (1999) · Journal of Obstetrics and Gynaecology · doi:10.1080/01443619964968 · Cited by 66
Journal Clubs During Advanced Pharmacy Practice Experiences to Teach Literature-Evaluation Skills
Arif, S. A., Gim, S., Nogid, A., & Shah, B. K. (2012) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe76588 · Cited by 65
How should individual participant data (IPD) from publicly funded clinical trials be shared?
Tudur Smith, C., Hopkins, C., Sydes, M. R., Woolfall, K., Clarke, M., Murray, G., & Williamson, P. (2015) · BMC Medicine · doi:10.1186/s12916-015-0532-z · Cited by 64
Sellig, C. B. (1991) · Journal of General Internal Medicine · doi:10.1007/bf02597431 · Cited by 62
Social media, medicine and the modern journal club
Topf, J. M., & Hiremath, S. S. (2015) · International Review of Psychiatry · doi:10.3109/09540261.2014.998991 · Cited by 55
Sharing Individual Participant Data (IPD) within the Context of the Trial Reporting System (TRS)
Zarin, D. A., & Tse, T. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1001946 · Cited by 53
Teaching Research and Evidence-Based Practice Using a Service-Learning Approach
Balakas, K., & Sparks, L. (2010) · Journal of Nursing Education · doi:10.3928/01484834-20100831-07 · Cited by 52
Hróbjartsson, A., Boutron, I., Turner, L., Altman, D. G., & Moher, D. (2013) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000058 · Cited by 49
Emergency Medicine Journal Clubs
Jouriles, N. J., Cordell, W. H., Martin, D. R., Wolfe, R. E., Emerman, C. L., & Avery, A. (1996) · Academic Emergency Medicine · doi:10.1111/j.1553-2712.1996.tb03535.x · Cited by 49
An emerging consensus for open evaluation: 18 visions for the future of scientific publishing
Kriegeskorte, N., Walther, A., & Deca, D. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00094 · Cited by 48
What makes evidence-based journal clubs succeed?
Phillips, B. (2004) · BMJ Evidence-Based Medicine · doi:10.1136/ebm.9.2.36 · Cited by 48
The Proliferation of Scientific Literature: A Natural Process
Ziman, J. M. (1980) · Science · doi:10.1126/science.7367863 · Cited by 46
A survey of journal clubs in U.S. family practice residencies
Heiligman, R. M., & Wollitzer, A. O. (1987) · Academic Medicine · doi:10.1097/00001888-198711000-00010 · Cited by 44
Akhund, S., & Kadir, M. M. (2006) · BMC Medical Education · doi:10.1186/1472-6920-6-43 · Cited by 44
Misconduct by researchers and authors
Marcovitch, H. (2007) · Gaceta Sanitaria · doi:10.1157/13112245 · Cited by 43
The role of the journal club in surgical residency programs: a survey of APDS program directors
Crank-Patton, A., Fisher, J. B., & Toedter, L. J. (2001) · Current Surgery · doi:10.1016/s0149-7944(00)00410-4 · Cited by 42
Fostering skills for evidence-based practice: The student journal club
Thompson, C. J. (2005) · Nurse Education in Practice · doi:10.1016/j.nepr.2005.08.004 · Cited by 42
Striving for evidence-based practice innovations through a hybrid model journal club: A pilot study
Wilson, M., Ice, S., Nakashima, C., Cox, L. A., Morse, E. C., Philip, G. H., & Vuong, E. (2015) · Nurse Education Today · doi:10.1016/j.nedt.2015.01.026 · Cited by 42
The Vacuum Shouts Back: Postpublication Peer Review on Social Media
Faulkes, Z. (2014) · Neuron · doi:10.1016/j.neuron.2014.03.032 · Cited by 42
Evaluation of Evidence-Based Methods Used to Teach Nursing Students to Critically Appraise Evidence
Smith‐Strøm, H., & Nortvedt, M. W. (2008) · Journal of Nursing Education · doi:10.3928/01484834-20080801-08 · Cited by 42
The Role of Journal Clubs in Orthopaedic Surgery Residency Programs
Greene, W. B. (2000) · Clinical Orthopaedics and Related Research · doi:10.1097/00003086-200004000-00037 · Cited by 39
The prevalence of underpowered randomized clinical trials in rheumatology.
Keen, H. I., Pile, K. D., & Hill, C. L. (2005) · PubMed · Cited by 38
What is the role of ‘the letter to the editor’?
Tierney, E., O’Rourke, C., & Fenton, J. E. (2014) · European Archives of Oto-Rhino-Laryngology · doi:10.1007/s00405-014-3289-7 · Cited by 37
Instructional Scaffolding to Improve Students’ Skills in Evaluating Clinical Literature
Dawn, S., Dominguez, K. D., Troutman, W. G., Bond, R., & Cone, C. J. (2011) · American Journal of Pharmaceutical Education · doi:10.5688/ajpe75462 · Cited by 36
Elnicki, D. M., Halperin, A. K., Shockcor, W. T., & Aronoff, S. C. (1999) · The American Journal of the Medical Sciences · doi:10.1097/00000441-199904000-00006 · Cited by 36
Black, C. (1985) · Cited by 36
Critically Appraised Topics (CATs)
Foster, N. E., Barlas, P., Chesterton, L. S., & Wong, J. (2001) · Physiotherapy · doi:10.1016/s0031-9406(05)60604-9 · Cited by 35
Journal clubs and case conferences: From academic tradition to communities of practice
Price, D. W., & Felix, K. (2008) · Journal of Continuing Education in the Health Professions · doi:10.1002/chp.180 · Cited by 35
Tutorials in Clinical Research, Part VIII: Creating a Journal Club
Forsen, J. W., Hartman, J. M., & Neely, J. G. (2003) · The Laryngoscope · doi:10.1097/00005537-200303000-00016 · Cited by 35
Sharing clinical trial data: a proposal from the International Committee of Medical Journal Editors
Taichman, D. B., Backus, J., Baethge, C., Bauchner, H., de Leeuw, P. W., Drazen, J. M., Fletcher, J., Frizelle, F., Groves, T., Haileamlak, A., et al. (2016) · The Lancet · doi:10.1016/s0140-6736(15)01279-9 · Cited by 35
Nursing research journal clubs: bridging the gap between practice and research.
Luby, M., Riley, J. K., & Towne, G. (2006) · PubMed · Cited by 35
Wooding, S., Pollitt, A., Castle‐Clarke, S., Cochrane, G., Diepeveen, S., Guthrie, S., Horvitz-Lennon, M. V., Larivière, V., Jones, M. M., Chonaill, S. N., et al. (2013) · Rand health quarterly · doi:10.7249/rhq-04-01-08 · Cited by 35
Poor-Quality Medical Research: What Can Journals Do?
Altman, D. G. (2002) · JAMA · doi:10.1001/jama.287.21.2765 · Cited by 34
Fetters, L., Figueiredo, E. M., Keane-Miller, D., McSweeney, D. J., & Tsao, C.-C. (2004) · Pediatric Physical Therapy · doi:10.1097/01.pep.0000114628.87352.c8 · Cited by 34
Bednarczyk, J. M., Pauls, M. A., Fridfinnson, J. A., & Weldon, E. (2014) · BMC Medical Education · doi:10.1186/1472-6920-14-57 · Cited by 34
Evidence-Based Orthopaedics: A Primer
Schünemann, H. J., & Bone, L. B. (2003) · Clinical Orthopaedics and Related Research · doi:10.1097/01.blo.0000080541.81794.26 · Cited by 34
Europe, R. (2011) · PsycEXTRA Dataset · doi:10.1037/e526292012-001 · Cited by 31
From bench-top to chair-side: How scientific evidence is incorporated into clinical practice
Tracy, S. L. (2013) · Dental Materials · doi:10.1016/j.dental.2013.08.200 · Cited by 28
Yarkoni, T. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00072 · Cited by 27
Proceedings of the conference on the postgraduate medical centre. Journal clubs
Mattingly, D. (1966) · Postgraduate Medical Journal · doi:10.1136/pgmj.42.484.120 · Cited by 26
Evidence-Based Nursing Leadership
Duffy, J. R., Thompson, D. J., Hobbs, T., Niemeyer-Hackett, N. L., & Elpers, S. (2011) · JONA The Journal of Nursing Administration · doi:10.1097/nna.0b013e31822edda6 · Cited by 26
Lizarondo, L. M., Grimmer, K. A., & Kumar, S. (2011) · BMC Medical Education · doi:10.1186/1472-6920-11-66 · Cited by 26
Learning from open source software projects to improve scientific review
Ghosh, S. S., Klein, A., Avants, B., & Millman, K. J. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00018 · Cited by 26
Implementing a Virtual Journal Club in a Clinical Nursing Setting
Berger, J., Hardin, H. K., & TOPP, R. V. (2011) · Journal for Nurses in Staff Development · doi:10.1097/nnd.0b013e318217b3bc · Cited by 25
Blogs and Twitter in medical publications: too unreliable to quote, or a change waiting to happen?
Chatterjee, P., & Biswas, T. (2011) · PubMed · Cited by 25
Repeat Revascularization Is a Faulty End Point for Clinical Trials
Kazi, D. S., & Hlatky, M. A. (2012) · Circulation Cardiovascular Quality and Outcomes · doi:10.1161/circoutcomes.112.965764 · Cited by 24
Teaching critical appraisal and statistics in anesthesia journal club
Moharari, R. S., Rahimi, E., Najafi, A., Khashayar, P., Khajavi, M., & Ap, M. (2008) · QJM · doi:10.1093/qjmed/hcn131 · Cited by 23
Pato, M. T., Cobb, R. T., Lusskin, S. I., & Schardt, C. M. (2013) · International Review of Psychiatry · doi:10.3109/09540261.2013.793172 · Cited by 22
Aggregating post-publication peer reviews and ratings
Florian, R. V. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00031 · Cited by 22
Julé, A. M., Vaillant, M., Lang, T. A., Guerin, P. J., & Olliaro, P. (2016) · PLoS neglected tropical diseases · doi:10.1371/journal.pntd.0004784 · Cited by 22
Online journal clubs: an innovative approach to achieving evidence-based practice.
Lizarondo, L. M., Kumar, S., & Grimmer, K. A. (2010) · PubMed · Cited by 21
EBM journal clubs in general practice.
Doust, J. A., Mar, C. D., Montgomery, B. D., Heal, C. F., Bidgood, R., Jeacocke, D., Bourke, G., & Spurling, G. (2008) · PubMed · Cited by 21
Promoting Evidence-Based Practice Through a Traveling Journal Club
Campbell-Fleming, J. M., Catania, K., & Courtney, L. (2008) · Clinical Nurse Specialist · doi:10.1097/01.nur.0000343077.73085.91 · Cited by 21
A journal club as a teaching and learning strategy in nurse teacher education
Sheehan, J. (1994) · Journal of Advanced Nursing · doi:10.1111/j.1365-2648.1994.tb01123.x · Cited by 20
Oliver, K. B., Dalrymple, P. W., Lehmann, H. P., McClellan, D. A., Robinson, K. A., & Twose, C. (2008) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.96.1.50 · Cited by 20
Setting Up, Maintaining and Evaluating an Evidence Based Radiology Journal Club
Kelly, A. M., & Cronin, P. (2010) · Academic Radiology · doi:10.1016/j.acra.2010.04.021 · Cited by 20
Leung, E., Tirlapur, S., Siassakos, D. M., & Khan, K. S. (2013) · BJOG An International Journal of Obstetrics & Gynaecology · doi:10.1111/1471-0528.12197 · Cited by 19
FOSE: a framework for open science evaluation
Walther, A., & van den Bosch, J. J. F. (2012) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2012.00032 · Cited by 19
On-the-Job Evidence-Based Medicine Training for Clinician-Scientists of the Next Generation.
Leung, E. Y. L., Malick, S. M., & Khan, K. S. (2013) · Europe PMC (PubMed Central) · Cited by 18
Teaching evidence-based medicine using team-based learning in journal clubs
Chakraborti, C. (2011) · Medical Education · doi:10.1111/j.1365-2923.2011.03972.x · Cited by 18
The importance of the journal club for neurosurgical trainees
Mobbs, R. J. (2003) · Journal of Clinical Neuroscience · doi:10.1016/s0967-5868(03)00074-2 · Cited by 18
Meta-Research: Broadening the Scope of PLOS Biology
Kousta, S. T., Ferguson, C., & Ganley, E. (2016) · PLoS Biology · doi:10.1371/journal.pbio.1002334 · Cited by 18
Bowden, S. C., Harrison, E., & Loring, D. W. (2013) · The Clinical Neuropsychologist · doi:10.1080/13854046.2013.776636 · Cited by 17
Translating Evidence-Based Decision Making into Practice: Appraising and Applying the Evidence
Miller, S. A., & Forrest, J. L. (2009) · Journal of Evidence Based Dental Practice · doi:10.1016/j.jebdp.2009.05.001 · Cited by 17
Pearce‐Smith, N. (2006) · Health Information & Libraries Journal · doi:10.1111/j.1471-1842.2006.00638.x · Cited by 16
Daily journal club: an education tool in palliative care
Mazuryk, M., Daeninck, P. J., Neumann, C. M., & Bruera, E. D. (2002) · Palliative Medicine · doi:10.1191/0269216302pm517oa · Cited by 16
The Returns from Arthritis Research Volume 1: Approach, Analysis and Recommendations
Wooding, S., Hanney, S. R., Buxton, I. L. O., & Grant, J. (2004) · Defense Technical Information Center (DTIC) · Cited by 14
Twitter-Based Journal Clubs: Additional Facts and Clarifications
Topf, J. M., Sparks, M. A., Iannuzzella, F., Lerma, E. V., Oates, T. M., Phelan, P. J., & Hiremath, S. S. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4639 · Cited by 14
Journal Clubs — Science as Conversation
Wright, J. (2004) · New England Journal of Medicine · doi:10.1056/nejmp048043 · Cited by 14
Implementing a successful journal club in an anesthesiology residency program
Pitner, N. D., Fox, C. A., & Riess, M. L. (2013) · F1000Research · doi:10.12688/f1000research.2-15.v1 · Cited by 14
Millichap, J. G., & Goldstein, J. L. (2011) · Neurology · doi:10.1212/wnl.0b013e31822b00f0 · Cited by 14
What makes evidence-based journal clubs succeed?
Robert S. Phillips, M., & Paul Glasziou, M. (2004) · ACP Journal Club · doi:10.7326/acpjc-2004-140-3-a11 · Cited by 14
Acquisition of Evidence-Based Surgery Skills in Plastic Surgery Residency Training
Temple, C. L. F., & Ross, D. C. (2011) · Journal of surgical education · doi:10.1016/j.jsurg.2010.12.004 · Cited by 13
Using Journal Clubs to Promote Skills for Evidence-Based Practice
Stern, P. (2008) · Occupational Therapy In Health Care · doi:10.1080/07380570802244381 · Cited by 13
The transformation of an EBM curriculum: a 10-year experience
Aiyer, M., & Dorsch, J. L. (2008) · Medical Teacher · doi:10.1080/01421590701881632 · Cited by 13
Redrawing the frontiers in the age of post-publication review
Galbraith, D. W. (2015) · Frontiers in Genetics · doi:10.3389/fgene.2015.00198 · Cited by 12
Dreier, M., Borutta, B., Stahmeyer, J. T., Krauth, C., & Walter, U. (2010) · PubMed Central · doi:10.3205/hta000085 · Cited by 12
Tam, K., Tsai, L., Wu, C., Wei, P., Wei, C., & Chen, S. (2011) · Journal of Evaluation in Clinical Practice · doi:10.1111/j.1365-2753.2011.01711.x · Cited by 12
Nursing Journal Clubs and the Clinical Nurse Specialist
Westlake, C. A., Albert, N. M., Rice, K. L., Bautista, C. A., Close, J., Foster, J., & Timmerman, G. M. (2014) · Clinical Nurse Specialist · doi:10.1097/nur.0000000000000095 · Cited by 12
Good, D. J., & McIntyre, C. M. (2015) · Journal of Nutrition Education and Behavior · doi:10.1016/j.jneb.2015.05.003 · Cited by 12
Making the most of peer review
Slavov, N. (2015) · eLife · doi:10.7554/elife.12708 · Cited by 12
Surgical Journal Clubs: A Bridge Connecting Experiential Learning Theory to Clinical Practice
Phitayakorn, R., Gelula, M. H., & Malangoni, M. A. (2006) · Journal of the American College of Surgeons · doi:10.1016/j.jamcollsurg.2006.09.018 · Cited by 11
Journal clubs for continued professional development
Grant, M. J. (2003) · Health Information & Libraries Journal · doi:10.1046/j.1365-2532.20.s1.11.x · Cited by 11
An Evidence‐Based Journal Club for Dental Residents in a GPR Program
Grant, W. D. (2005) · Journal of Dental Education · doi:10.1002/j.0022-0337.2005.69.6.tb03951.x · Cited by 11
How to Perform a Critically Appraised Topic: Part 2, Appraise, Evaluate, Generate, and Recommend
Kelly, A. M., & Cronin, P. (2011) · American Journal of Roentgenology · doi:10.2214/ajr.11.7249 · Cited by 11
Taking critical appraisal to extremes. The need for balance in the evaluation of evidence.
Woolf, S. H. (2000) · PubMed · Cited by 11
Landi, M., Springer, S. P., Estus, E. L., & Ward, K. E. (2015) · The Consultant Pharmacist · doi:10.4140/tcp.n.2015.356 · Cited by 10
Ibrahim, A., Mshelbwala, P., Mai, A., Asuku, M., & Mbibu, H. N. (2014) · Nigerian Journal of Surgery · doi:10.4103/1117-6806.137292 · Cited by 10
Laibhen‐Parkes, N. (2013) · Journal of Pediatric Nursing · doi:10.1016/j.pedn.2013.10.003 · Cited by 10
When postpublication peer review stings
Collier, R. (2014) · Canadian Medical Association Journal · doi:10.1503/cmaj.109-4817 · Cited by 10
Setting up a multidisciplinary journal club in learning disability
Hunt, C., & Topham, L. (2002) · British Journal of Nursing · doi:10.12968/bjon.2002.11.10.688 · Cited by 10
Wolinsky, H. (2011) · EMBO Reports · doi:10.1038/embor.2011.201 · Cited by 10
A New Capability: Postpublication Peer Review forPediatrics
Anderson, K. R. (1999) · PEDIATRICS · doi:10.1542/peds.104.1.106 · Cited by 9
Role of journal clubs in undergraduate medical education
Biswas, T. (2011) · Indian Journal of Community Medicine · doi:10.4103/0970-0218.91422 · Cited by 9
Enlivening journal clubs using a modified ‘jigsaw’ method
Willett, L. R., Kim, S., & Gochfeld, M. (2013) · Medical Education · doi:10.1111/medu.12342 · Cited by 9
Hinkson, C. R., Kaur, N., Sipes, M. W., & Pierson, D. J. (2011) · Respiratory Care · doi:10.4187/respcare.00952 · Cited by 9
Traditional journal club: a continuing problem.
Misra, U. K., Kalita, J., & Nair, P. (2007) · PubMed · Cited by 9
Shokouhi, G., Ghojazadeh, M., & Sattarnezhad, N. (2012) · International Journal of Health Sciences · doi:10.12816/0005973 · Cited by 8
Austin, T. M., Richter, R. R., & Frese, T. (2009) · Physiotherapy Research International · doi:10.1002/pri.444 · Cited by 8
Bachmann, T. (2011) · Frontiers in Computational Neuroscience · doi:10.3389/fncom.2011.00061 · Cited by 8
Journal clubs on the night shift: a staff nurse initiative.
Stewart, C., Snyder, K., & Sullivan, S. C. (2011) · PubMed · Cited by 7
Planning and executing orthopedic journal clubs
Moro, J. K., & Bhandari, M. (2007) · Indian Journal of Orthopaedics · doi:10.4103/0019-5413.30525 · Cited by 7
Empowering change with traditional or virtual journal clubs
Dovi, G. (2015) · Nursing Management · doi:10.1097/01.numa.0000459699.28739.c9 · Cited by 7
Journal clubs most effective if tailored to learner needs
Matthews, D. (2011) · Evidence-Based Dentistry · doi:10.1038/sj.ebd.6400818 · Cited by 6
One Model of Residency Journal Clubs With Multifaceted Support
Serghi, A., Goebert, D. A., Andrade, N. N., Hishinuma, E. S., Lunsford, R. M., & Matsuda, N. M. (2015) · Teaching and Learning in Medicine · doi:10.1080/10401334.2015.1044658 · Cited by 6
Critically Appraised Topics (CATs)
Shannon, S. (2008) · Archives of Dermatology · doi:10.1001/archderm.144.10.1327 · Cited by 6
Response to “Twitter-Based Journal Clubs: Some Additional Facts and Clarifications”
Perera, M., Roberts, M. J., Lawrentschuk, N., & Bolton, D. M. (2015) · Journal of Medical Internet Research · doi:10.2196/jmir.4786 · Cited by 5
(2012) · Nature Medicine · doi:10.1038/nm.2978 · Cited by 5
Journal clubs in clinical medicine
Jones, S. (2002) · Emergency Medicine Journal · doi:10.1136/emj.19.2.184-b · Cited by 4
(2009) · Nature · doi:10.1038/460551a · Cited by 4
Journal Club Experience in a Postgraduate Psychiatry Program in Chile
Figueroa, R. A., Valdivieso, S., Turpaud, M., Cortes, P., de Barros, J., & Castano, C. (2009) · Academic Psychiatry · doi:10.1176/appi.ap.33.5.407 · Cited by 3
Referees make journal clubs fun
Gonzalez, L. S. (2003) · BMJ · doi:10.1136/bmj.326.7380.106 · Cited by 3
Young Investigator Perspectives. Blogging for electronic record keeping and collaborative research
Sandefur, C. I. (2014) · American Journal of Physiology-Gastrointestinal and Liver Physiology · doi:10.1152/ajpgi.00384.2014 · Cited by 2
Getting paediatric clinical trials published
Smyth, R. L. (2016) · The Lancet · doi:10.1016/s0140-6736(16)32125-0 · Cited by 2
Journal clubs--knowledge sufficient for critical appraisal of the literature?
Rheeder, P., Zyl, D. G. V., Webb, E., & Worku, Z. (2007) · PubMed · Cited by 1
Journal Club: A venue to advance evidence-based infection prevention practice
Manning, M. L., & Davis, J. (2012) · American Journal of Infection Control · doi:10.1016/j.ajic.2012.07.001 · Cited by 1
Lloyd, G. W. (1999) · Emergency Medicine Journal · doi:10.1136/emj.16.3.238-c · Cited by 1
Research works printed but not linked: 1
- Cortese, 2007, The Learning Healthcare System: Workshop Summary
Cited by
Tod, D., Booth, A., & Smith, B. M. (2021) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2021.1952471 · Cited by 169
“Critical appraisal is performed in a series of steps so that reviewers complete the task in a systematic and consistent fashion (Goldstein et al., 2017; Tod, 2019).”
AI-generated text may have a role in evidence-based medicine
Peng, Y., Rousseau, J. F., Shortliffe, E. H., & Weng, C. (2023) · Nature Medicine · doi:10.1038/s41591-023-02366-9 · Cited by 64
Self-citationZhao, D., Zhu, G., Ding, Y., & Zheng, J. (2018) · Polymers · doi:10.3390/polym10070716 · Cited by 50
The voices of children and young people during COVID ‐19: A critical review of methods
Jörgensen, E., Koller, D., Raman, S., Olatunya, O. S., Asemota, O. A., Ekpenyong, B. N., Gunnlaugsson, G., & Okolo, A. A. (2022) · Acta Paediatrica · doi:10.1111/apa.16422 · Cited by 26
Understanding the nature and scope of clinical research commentaries in PubMed
Rogers, J. R., Mills, H., Liu, L. G., Goldstein, A., & Weng, C. (2019) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocz209 · Cited by 19
“Evidence appraisals are communicated in a variety of formats.(6) In person meetings, particularly in the form of journal clubs, promote active face-to-face discussion.”
BackgroundSelf-citation
Metadata
- Authors
- Goldstein, A., Venker, E., & Weng, C.
- Affiliations
- Columbia University (US)
- Corresponding author
- Goldstein, Andrew
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2017-05-24
- Volume
- 24
- Issue
- 6
- Pages
- 1192-1203
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6259661
- OpenAlex
- W2620100062
- Cited by
- 30
- References
- 197
- Citations by year
- 2026: 1 · 2025: 2 · 2024: 5 · 2023: 8 · 2022: 3 · 2021: 2 · 2020: 2 · 2019: 4 · 2018: 3
- FWCI
- 0.61
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- other-oa
- Locations
- Journal of the American Medical Informatics Association (publishedVersion, other-oa)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- U.S. National Library of Medicine (R01LM009886)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- evidence appraisal, critical appraisal, clinical evidence, conceptual framework, research validity, research utilization, knowledge translation, assessment instruments
- MeSH terms
- Medical Informatics, Periodicals as Topic, Publishing, Peer Review, Research, Evidence-Based Medicine, Translational Research, Biomedical
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.