The DOI matches its Crossref record; 28 of the 28 references it cites were found in a registry or an index.
Reproducibility of Literature Search Reporting in Medical Education Reviews
Abstract
PURPOSE: Medical education literature has been found to lack key components of scientific reporting, including adequate descriptions of literature searches, thus preventing medical educators from replicating and building on previous scholarship. The purpose of this study was to examine the reproducibility of search strategies as reported in medical education literature reviews. METHOD: The authors searched for and identified literature reviews published in 2009 in Academic Medicine, Teaching and Learning in Medicine, and Medical Education. They searched for citations whose titles included the words "meta-analysis," "systematic literature review," "systematic review," or "literature review," or whose publication type MEDLINE listed as "meta-analysis" or "review." The authors created a checklist to identify key characteristics of literature searches and of literature search reporting within the full text of the reviews. The authors deemed searches reproducible only if the review reported both a search date and Boolean operators. RESULTS: Of the 34 reviews meeting the inclusion criteria, 19 (56%) explicitly described a literature search and mentioned MEDLINE; however, only 14 (41%) also mentioned searches of nonmedical databases. Eighteen reviews (53%) listed search terms, but only 6 (18%) listed Medical Subject Headings, and only 2 (6%) mentioned Boolean operators. Fifteen (44%) noted the use of limits. None of the reviews included reproducible searches. CONCLUSIONS: According to this analysis, literature search strategies in medical education reviews are highly variable and generally not reproducible. The authors provide recommendations to facilitate future high-quality, transparent, and reproducible literature searches.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“Researchers assessing the reproducibility of the search have used varying methods to determine what constitutes a reproducible search [11, 17, 19, 20].”
From PRISMA-S: an extension to the PRISMA Statement for Reporting Literature Searches in Systematic Reviews (2021) · cites this paper as [19]
Citations
Cites
Liberati, A., Altman, D. G., Tetzlaff, J., Mulrow, C., Gøtzsche, P. C., Ioannidis, J. P. A., Clarke, M., Devereaux, P. J., Kleijnen, J., & Moher, D. (2009) · PLoS Medicine · doi:10.1371/journal.pmed.1000100 · Cited by 28,164
Virtual patients: a critical literature review and proposed next steps
Cook, D. A., & Triola, M. M. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03286.x · Cited by 624
An evidence-based practice guideline for the peer review of electronic search strategies
Sampson, M., McGowan, J., Cogo, E., Grimshaw, J., Moher, D., & Lefebvre, C. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2008.10.012 · Cited by 608
Cook, D. A., Bordage, G., & Schmidt, H. G. (2008) · Medical Education · doi:10.1111/j.1365-2923.2007.02974.x · Cited by 472
Lowe, H. J. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03510380059038 · Cited by 468
Systematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257
Jha, V., Quinton, N. D., Bekker, H. L., & Roberts, T. E. (2009) · Medical Education · doi:10.1111/j.1365-2923.2008.03244.x · Cited by 242
Uniform Requirements for Manuscripts Submitted to Biomedical Journals
(1997) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1997.03540350077040 · Cited by 242
A systematic review of resident‐as‐teacher programmes
Hill, A. G., Yu, T., Barrow, M., & Hattie, J. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03523.x · Cited by 230
Sociological interpretations of professionalism
Martimianakis, M. A., Maniate, J. M., & Hodges, B. D. (2009) · Medical Education · doi:10.1111/j.1365-2923.2009.03408.x · Cited by 211
Research works printed but not linked: 3
- Haig, 2003, Med Teach, 25, 352
- McGowan, 2005, J Med Libr Assoc, 93, 74
- 1997, JAMA, 277, 927, 10.1001/jama.277.11.927 · doi:10.1001/jama.277.11.927 · Paper page
Cited by
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., & Koffel, J. (2019) · doi:10.31219/osf.io/sfc38 · Cited by 3,696
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., PRISMA-S Group, Blunt, H., Brigham, T., et al. (2021) · Systematic Reviews · doi:10.1186/s13643-020-01542-z · Cited by 3,714
Cooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Rethlefsen, M. L., Kirtley, S., Waffenschmidt, S., Ayala, A. P., Moher, D., Page, M. J., Koffel, J. B., & Group, P.-S. (2021) · Journal of the Medical Library Association · doi:10.5195/jmla.2021.962 · Cited by 409
Rethlefsen, M. L., Farrell, A. M., Osterhaus Trzasko, L. C., & Brigham, T. J. (2015) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2014.11.025 · Cited by 331
Metadata
- Authors
- Maggio, L. A., Tannery, N. H., & Kanter, S. L.
- Affiliations
- Stanford Medicine (US)
- Stanford University (US)
- Corresponding author
- Maggio, Lauren A.
- Venue
- Academic Medicine
- Publisher
- Oxford University Press (OUP)
- Published
- 2011-08
- Volume
- 86
- Issue
- 8
- Pages
- 1049-1054
- ISSN
- 1040-2446 (ISSN-L)
- OpenAlex
- W1992951084
- Cited by
- 77
- References
- 28
- Citations by year
- 2026: 3 · 2025: 2 · 2024: 7 · 2023: 8 · 2022: 4 · 2021: 14 · 2020: 7 · 2019: 2 · 2018: 4 · 2017: 3 · 2016: 5 · 2015: 4 · 2014: 5 · 2013: 5 · 2012: 4
- FWCI
- 4.77
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Academic Medicine (publishedVersion)
- PubMed (publishedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › Academic Writing and Publishing
- Social Sciences › Education › Innovations in Medical Education
- Keywords
- reproducibility, search strategies, medical education research, systematic reviews, meta-analysis, Boolean operators, Medical Subject Headings, search limits, research transparency
- MeSH terms
- Education, Medical, Humans, Review Literature as Topic, Reproducibility of Results, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reproducibility of Literature Search Reporting in Medical Education Reviews. https://app.cochat.ai/papers/10.1097/acm.0b013e31822221e7
- The paper
- Maggio, L. A., Tannery, N. H., & Kanter, S. L. (2011). Reproducibility of Literature Search Reporting in Medical Education Reviews. Academic Medicine. https://doi.org/10.1097/acm.0b013e31822221e7
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.