Reproducibility of Literature Search Reporting in Medical Education Reviews
The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Abstract
PURPOSE: Medical education literature has been found to lack key components of scientific reporting, including adequate descriptions of literature searches, thus preventing medical educators from replicating and building on previous scholarship. The purpose of this study was to examine the reproducibility of search strategies as reported in medical education literature reviews. METHOD: The authors searched for and identified literature reviews published in 2009 in Academic Medicine, Teaching and Learning in Medicine, and Medical Education. They searched for citations whose titles included the words "meta-analysis," "systematic literature review," "systematic review," or "literature review," or whose publication type MEDLINE listed as "meta-analysis" or "review." The authors created a checklist to identify key characteristics of literature searches and of literature search reporting within the full text of the reviews. The authors deemed searches reproducible only if the review reported both a search date and Boolean operators. RESULTS: Of the 34 reviews meeting the inclusion criteria, 19 (56%) explicitly described a literature search and mentioned MEDLINE; however, only 14 (41%) also mentioned searches of nonmedical databases. Eighteen reviews (53%) listed search terms, but only 6 (18%) listed Medical Subject Headings, and only 2 (6%) mentioned Boolean operators. Fifteen (44%) noted the use of limits. None of the reviews included reproducible searches. CONCLUSIONS: According to this analysis, literature search strategies in medical education reviews are highly variable and generally not reproducible. The authors provide recommendations to facilitate future high-quality, transparent, and reproducible literature searches.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Researchers assessing the reproducibility of the search have used varying methods to determine what constitutes a reproducible search [11, 17, 19, 20].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [19]
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Virtual patients: a critical literature review and proposed next steps
Cook, D. A., & Triola, M. M. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03286.x · Cited by 624
An evidence-based practice guideline for the peer review of electronic search strategies
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.012 · Cited by 608
Cook, D. A., Bordage, G., & Schmidt, H. G. (2008) · Medical Education · doi:10.1111/j.1365-2923.2007.02974.x · Cited by 472
Lowe, H. J. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03510380059038 · Cited by 468
Systematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257
Jha, V., Quinton, N. D., Bekker, H. L., & Roberts, T. E. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03244.x · Cited by 242
Uniform Requirements for Manuscripts Submitted to Biomedical Journals
(1997) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1997.03540350077040 · Cited by 242
A systematic review of resident‐as‐teacher programmes
Hill, A. G., Yu, T., Barrow, M., & Hattie, J. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03523.x · Cited by 230
Sociological interpretations of professionalism
Martimianakis, M. A., Maniate, J. M., & Hodges, B. D. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03408.x · Cited by 211
Research works printed but not linked: 3
- Haig, 2003, Med Teach, 25, 352
- McGowan, 2005, J Med Libr Assoc, 93, 74
- 1997, JAMA, 277, 927, 10.1001/jama.277.11.927 · doi:10.1001/jama.277.11.927 · Paper page
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.962 · Cited by 409
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
Ten Simple Rules for Writing a Literature Review
Pautasso, M. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003149 · Cited by 272
On the reproducibility of meta-analyses: six practical recommendations
Lakens, D., Hilgard, J., & Staaks, J. (2016) · BMC Psychology · doi:10.1186/s40359-016-0126-3 · Cited by 230
Boeker, M., Vach, W., & Motschall, E. (2013) · BMC Medical Research Methodology · doi:10.1186/1471-2288-13-131 · Cited by 225
“Background For many scientists, especially in the life-science domains, the literature retrieval process is tedious, errorprone and nontransparent [1-4].”
BackgroundConducting systematic reviews in medical education: a stepwise approach
Cook, D. A., & West, C. P. (2012) · Medical Education · doi:10.1111/j.1365-2923.2012.04328.x · Cited by 186
Koffel, J. B. (2015) · PLOS ONE · doi:10.1371/journal.pone.0125931 · Cited by 159
The Structure and Conduct of a Narrative Literature Review
Pautasso, M. (2019) · A Guide to the Scientific Career · doi:10.1002/9781118907283.ch31 · Cited by 124
Koffel, J. B., & Rethlefsen, M. L. (2016) · PLOS ONE · doi:10.1371/journal.pone.0163309 · Cited by 107
López‐Nicolás, R., López-López, J. A., Rubio‐Aparicio, M., & Sánchez‐Meca, J. (2021) · Behavior Research Methods · doi:10.3758/s13428-021-01644-z · Cited by 80
Rethlefsen, M. L., Brigham, T., Price, C. L., Moher, D., Bouter, L. M., Kirkham, J. J., Schroter, S., & Zeegers, M. P. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.111229 · Cited by 77
Conducting a successful systematic review of the literature, part 1
Riesenberg, L. A., & Justice, E. M. (2014) · Nursing · doi:10.1097/01.nurse.0000444728.68018.ac · Cited by 63
Page, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S., Haddaway, N., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01670-0 · Cited by 63
Toews, L. C. (2017) · Journal of the Medical Library Association · doi:10.5195/jmla.2017.246 · Cited by 58
Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews
Golder, S., Loke, Y. K., & Zorzela, L. (2014) · Health Information & Libraries Journal · doi:10.1111/hir.12041 · Cited by 55
Papageorgiou, S. N., Papadopoulos, M. A., & Athanasiou, A. E. (2014) · The European Journal of Orthodontics · doi:10.1093/ejo/cjt008 · Cited by 51
Games to support teaching clinical reasoning in health professions education: a scoping review
Koelewijn, G. R., Hennus, M. P., Kort, H. S. M., Frenkel, J., & van Houwelingen, T. (2024) · Medical Education Online · doi:10.1080/10872981.2024.2316971 · Cited by 51
Writing for academia: Getting your research into print: AMEE Guide No. 74
Coverdale, J., Roberts, L. W., Balon, R., & Beresin, E. V. (2012) · Medical Teacher · doi:10.3109/0142159x.2012.742494 · Cited by 49
An Inadvertent Concurrent Replication: Same Roadmap, Different Journey
Lemons, C. J., King, S. A., Davidson, K. A., Berryessa, T. L., Gajjar, S. A., & Sacks, L. H. (2016) · Remedial and Special Education · doi:10.1177/0741932516631116 · Cited by 48
Evaluating Article Search and Selection Procedures in Special Education Literature Reviews
King, S. A., Davidson, K. A., Chitiyo, A., & Apple, D. (2018) · Remedial and Special Education · doi:10.1177/0741932518813142 · Cited by 48
Golder, S., Loke, Y. K., & Zorzela, L. (2013) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2012.09.013 · Cited by 39
Educating physicians in evidence based medicine: current practices and curricular strategies
Maggio, L. A. (2016) · Perspectives on Medical Education · doi:10.1007/s40037-016-0301-5 · Cited by 39
Self-citationA suggested data structure for transparent and repeatable reporting of bibliographic searching
Haddaway, N., Rethlefsen, M. L., Davies, M. C., Glanville, J., McGowan, B. S., Nyhan, K., & Young, S. (2022) · Campbell Systematic Reviews · doi:10.1002/cl2.1288 · Cited by 38
Wachtel, R. E., & Dexter, F. (2013) · Anesthesia & Analgesia · doi:10.1213/ane.0b013e3182a6d33b · Cited by 36
Oliván, J. A. S., Cuenca, G. M., & Avilés, R. A. (2018) · Revista española de Documentación Científica · doi:10.3989/redc.2018.2.1491 · Cited by 29
Substance use Education in US Schools of Pharmacy: A Systematic Review of the Literature
Muzyk, A. J., Peedin, E., Lipetzky, J., Parker, H., MacEachern, M., & Thomas, K. L. H. (2017) · Substance Abuse · doi:10.1080/08897077.2017.1341448 · Cited by 29
Literature search strategies in dental education and research
Khurshid, Z., Tariq, R., Asiri, F. Y. I., Abid, K., & Zafar, M. S. (2021) · Journal of Taibah University Medical Sciences · doi:10.1016/j.jtumed.2021.05.012 · Cited by 29
Conducting a successful systematic review of the literature, part 2
Riesenberg, L. A., & Justice, E. M. (2014) · Nursing · doi:10.1097/01.nurse.0000446641.02995.6a · Cited by 22
A new method for testing reproducibility in systematic reviews was developed, but needs more testing
Pieper, D., Heß, S., & Faggion, C. M. (2021) · BMC Medical Research Methodology · doi:10.1186/s12874-021-01342-6 · Cited by 21
Rethlefsen, M. L., Schroter, S., Bouter, L. M., Moher, D., Ayala, A. P., Kirkham, J. J., & Zeegers, M. P. (2021) · Trials · doi:10.1186/s13063-021-05738-z · Cited by 17
Quality of Literature Searches Published in Leading Neurosurgical Journals: A Review of Reviews
O’Donohoe, T. J., Bridson, T. L., Shafik, C. G., Wynne, D., Dhillon, R., & Tee, J. (2021) · Neurosurgery · doi:10.1093/neuros/nyaa573 · Cited by 15
Sparacio, A., Ropovik, I., Jiga‐Boy, G. M., Forscher, P. S., Paris, B., & IJzerman, H. R. (2021) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zpw28 · Cited by 14
Open reproducible scientometric research with Alexandria3k
Spinellis, D. D. (2023) · PLoS ONE · doi:10.1371/journal.pone.0294946 · Cited by 14
The role of medical librarians in medical education review articles
Tannery, N. H., & Maggio, L. A. (2012) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.100.2.015 · Cited by 13
Self-citationNeeleman, R., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02472-w · Cited by 12
Rylee, T. L., & Cavanagh, S. J. (2022) · Health Services and Outcomes Research Methodology · doi:10.1007/s10742-022-00270-2 · Cited by 12
Dix règles simples pour rédiger une revue de littérature
Pautasso, M. (2014) · Kinésithérapie la Revue · doi:10.1016/j.kine.2014.01.008 · Cited by 11
General practitioners training about suicide prevention and risk: A systematic review of literature
Audouard-Marzin, Y., Kopp-Bigault, C., Scouarnec, P., & Walter, M. (2019) · La Presse Médicale · doi:10.1016/j.lpm.2019.05.028 · Cited by 11
Rethlefsen, M. L., Schroter, S., Bouter, L. M., Kirkham, J. J., Moher, D., Ayala, A. P., Blanco, D., Brigham, T., Nardini, H. K. G., Kirtley, S., et al. (2025) · BMJ Evidence-Based Medicine · doi:10.1136/bmjebm-2024-113527 · Cited by 10
On the Use and Value of Reporting Guidelines in Health Professions Education Research
OʼBrien, B. C., West, C. P., Coverdale, J., Durning, S. J., & Roberts, L. W. (2020) · Academic Medicine · doi:10.1097/acm.0000000000003666 · Cited by 10
Stress Regulation via Being in Nature and Social Support in Adults, a Meta-analysis
Sparacio, A., Ropovik, I., Jiga‐Boy, G. M., Cem, A., & IJzerman, H. R. (2023) · Collabra Psychology · doi:10.1525/collabra.77343 · Cited by 9
Some Potential “Pitfalls” in the Construction of Educational Systematic Reviews
Coverdale, J., Roberts, L. W., Beresin, E. V., Louie, A. K., Brenner, A. M., & Balon, R. (2017) · Academic Psychiatry · doi:10.1007/s40596-017-0675-7 · Cited by 9
Reproducibility of Published Meta-Analyses on Clinical-Psychological Interventions
López‐Nicolás, R., Lakens, D., López-López, J. A., Rubio‐Aparicio, M., Sandoval‐Lentisco, A., López‐Ibáñez, C., Blázquez-Rincón, D., & Sánchez‐Meca, J. (2024) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459231202929 · Cited by 6
Systemic review and meta-analysis of diagnostic imaging technologies
Wáng, Y. X. J. (2016) · Quantitative Imaging in Medicine and Surgery · doi:10.21037/qims.2016.10.08 · Cited by 6
Revisión sistemática de la bibliografía (parte 1)
Riesenberg, L. A., & Justice, E. M. (2014) · Nursing (Ed española) · doi:10.1016/j.nursi.2014.12.019 · Cited by 6
The quality analysis of literature retrievals of systematic reviews for traditional Chinese medicine
Chen, M., Xiao, Y., Liu, Y., Peng, Y., He, J., Zhang, Y., & Du, L. (2015) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12139 · Cited by 5
Interpretations of reproducibility crisis in medical education research: a qualitative study
Ahmady, S., Kohan, N., Hamidi, H., Vahednasiri, A., & Hooshmandja, M. (2026) · Scientific Reports · doi:10.1038/s41598-025-34640-w · Cited by 5
Nurse-Led Advance Care Planning in Adults in the U.S.- A Scoping Review
Agyei, K. G., Yahaya, A., Dapilah, E., & Norton, S. A. (2024) · American Journal of Hospice and Palliative Medicine® · doi:10.1177/10499091241276009 · Cited by 5
Literature Searching in Medical Education: Online Tutorial Development from Idea to Creation
Maggio, L. A., Davies, K. J., Allee, N. J., Beattie, J., Berryman, D. R., Littleton, D. M., Tannery, N. H., & O’Rourke, K. (2012) · Medical Reference Services Quarterly · doi:10.1080/02763869.2012.724274 · Cited by 5
Self-citationKarakitsos, P., & Mylonas, K. S. (2024) · International Journal of Cardiology · doi:10.1016/j.ijcard.2024.131987 · Cited by 3
Lock, I., & Giani, S. (2021) · SocArXiv (OSF Preprints) · doi:10.31235/osf.io/8ek6h · Cited by 3
Fehrmann, P. G., & Mamolen, M. (2020) · IASSIST Quarterly · doi:10.29173/iq968 · Cited by 1
A meta-scientific approach to understanding stress regulation
Sparacio, A. (2023) · Cronfa (Swansea University) · doi:10.23889/suthesis.63454
A meta-scientific approach to understanding stress regulation
Sparacio, A. (2023) · theses.fr (ABES) · doi:10.70675/757862f2z87a4z4ec7zabc9z5a275e01261b
Rethlefsen, M. L. (2024) · Research Publications (Maastricht University) · doi:10.26481/dis.20250121mr
Rules for review of writing methodology
Bakhridinova, O., & Hakimov, Q. (2021) · Общество и инновации · doi:10.47689/2181-1415-vol2-iss6/s-pp67-71
Reporting Standards for Literature Reviews
Dekkers, R., Carey, L., & Langhorne, P. (2022) · Springer eBooks · doi:10.1007/978-3-030-90025-0_13
Rethlefsen, M., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. B. (2021) · Figshare · doi:10.6084/m9.figshare.13646921
Page, M. J., Moher, D., Fidler, F. M., Higgins, J. P. T., Brennan, S. E., Haddaway, N. R., Hamilton, D. G., Kanukula, R., Karunananthan, S., Maxwell, L. J., et al. (2021) · Figshare · doi:10.6084/m9.figshare.14442918
Communication and information needs of patients with cancer in Ghana: A scoping review
Agyei, K. G., Yahaya, A., Leblanc, N. M., Loh, K. P., & Norton, S. A. (2026) · PLoS ONE · doi:10.1371/journal.pone.0343094
Alizadeh, M., Rahmati, R., Zarimeidani, F., Hasani, F., Ghaedi, A., Bazrgar, A., Doalame, R. H., Vahedi, H. S. M., Hekmat, H., & Omidi, N. (2025) · Medicine · doi:10.1097/md.0000000000042316
On the Reproducibility of Meta-Analyses
Lakens, D. (2018) · doi:10.23668/psycharchives.2441
Neeleman, R. C., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R. (2024) · Figshare · doi:10.6084/m9.figshare.26682223
Assessing Computer Searches Used for Systematic Reviews
Fehrmann, P. G. (2015)
Probiotics Effect on the Immune SystemPreprint
Ng, J. C. K. (2020) · Preprints.org · doi:10.20944/preprints202006.0248.v1
Probiotics Effect on the Immune SystemPreprint
Ng, J. C. K. (2020) · Research Square · doi:10.21203/rs.3.rs-37274/v1
Probiotics Effect on the Immune SystemPreprint
Ng, J. C. K. (2020) · OSF Preprints (OSF Preprints) · doi:10.31219/osf.io/kahnp
Probiotics Effect on the Immune SystemPreprint
Ng, J. C. K. (2020) · Frenxiv (OSF Preprints) · doi:10.31226/osf.io/b3x5y
Bridging formalisation and expert judgement in searches for studies for systematic reviews
Briscoe, S. (2023) · Open Research Exeter (University of Exeter)
I.LOCK_S.GIANI_S1_SHORT PAPER_Review of reviews
Lock, I., & Giani, S. (2020) · Figshare · doi:10.21942/uva.12369266
Levin, L. L., & Lee, Y.-J. (2014) · The Journal of the American Medical Association (JAMA) Network (American Medical Association) · doi:10.13028/nwxy-dx38
Where critique can become collaboration: celebrating Academic Medicine
Maggio, L. A. (2026) · Academic Medicine · doi:10.1093/acamed/wvag223
Self-citation
Metadata
- Authors
- Maggio, L. A., Tannery, N. H., & Kanter, S. L.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Corresponding author
- Maggio, Lauren A.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-08
- Volume
- 86
- Issue
- 8
- Pages
- 1049-1054
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W1992951084
- Cited by
- 77
- References
- 27
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 7 · 2023: 8 · 2022: 4 · 2021: 14 · 2020: 7 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 5 · 2015: 4 · 2014: 5 · 2013: 5 · 2012: 4
- FWCI
- 4.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Academic Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- reproducibility, search strategies, medical education research, systematic reviews, meta-analysis, Boolean operators, Medical Subject Headings, search limits, research transparency
- MeSH terms
- Education, Medical, Humans, Review Literature as Topic, Reproducibility of Results, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.