Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
A complete description of the literature search, including the criteria used for the inclusion of reports after they have been located, used in a research synthesis or meta‐analysis is critical if subsequent researchers are to accurately evaluate and reproduce a synthesis' methods and results. Based on previous guidelines and new suggestions, we present a set of focused and detailed standards for reporting the methods used in a literature search. The guidelines cover five search strategies: reference database searches, journal and bibliography searches, searches of the reference lists of reports, citation searches, and direct contact searches. First, we bring together all the unique recommendations made in existing guidelines for research synthesis. Second, we identify gaps in reporting standards for search strategies. Third, we address these gaps by providing new reporting recommendations. Our hope is to facilitate successful evaluation and replication of research synthesis results. Copyright © 2014 John Wiley & Sons, Ltd.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Des Jarlais, D. C., Lyles, C., Crepaz, N., & the TREND Group (2004) · American Journal of Public Health · doi:10.2105/ajph.94.3.361 · Cited by 2,817
Lipsey, M. W., & Wilson, D. B. (1993) · American Psychologist · doi:10.1037/0003-066x.48.12.1181 · Cited by 2,033
Research Synthesis and Meta-Analysis: A Step-by-Step Approach
Cooper, H. (2017) · doi:10.4135/9781071878644 · Cited by 1,716
Self-citationPashler, H., & Wagenmakers, E. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612465253 · Cited by 1,321
Cook, D. J., Sackett, D. L., & Spitzer, W. O. (1995) · Journal of Clinical Epidemiology · doi:10.1016/0895-4356(94)00172-m · Cited by 573
Reporting standards for research in psychology: why do we need them? What might they be?
APA Publications and Communications Board Working Group on Journal Article Reporting Standards (2008) · American Psychologist · doi:10.1037/0003-066x.63.9.839 · Cited by 571
Why Science Is Not Necessarily Self-Correcting
Ioannidis, J. P. A. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612464056 · Cited by 563
Duval, S. (2005) · Publication Bias in Meta‐Analysis · doi:10.1002/0470870168.ch8 · Cited by 496 · cited 1×
“It is because of this that research syntheses often include tests for publication bias (Duval, 2005) and moderator analyses of publication status (if unpublished research is included).”
BackgroundGiner-Sorolla, R. (2012) · Perspectives on Psychological Science · doi:10.1177/1745691612457576 · Cited by 274
Research works printed but not linked: 16
- American Psychological Association, 2010, Publication Manual of the American Psychological Association
- 10.1177/107554708901000301 · DOI · Paper page
- 10.1186/cc3803 · DOI · Paper page
- Fink A, 2005, Conducting research literature reviews: From the internet to paper
- Galvan JL, 2006, Writing literature reviews
- 10.1371/journal.pone.0072467 · DOI · Paper page
- Hart C, 1998, Doing a literature review: Releasing the social science research imagination
- 10.1016/S0140-6736(99)04149-5 · DOI · Paper page
- 10.1371/journal.pmed.1000097 · DOI · Paper page
- 10.1371/journal.pmed.0040078 · DOI · Paper page
- OldsJ.2014.Replication problem workshop at NSF [Web log post]. Retrieved March 2014 fromhttp://krasnow.blogspot.com/2014/02/replication‐problem‐workshop‐at‐nsf.html
- Pan ML, 2008, Preparing literature reviews: Qualitative and quantitative approaches
- 10.1126/science.149.3683.510 · DOI · Paper page
- 10.1111/1467-9280.00169 · DOI · Paper page
- 10.1001/jama.283.15.2008 · DOI · Paper page
- W.T. Grant Foundation, 2013, Portfolio of Grants Related to Education
Cited by
Gusenbauer, M., & Haddaway, N. R. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1378 · Cited by 2,379
“We do not test the searchers' ability to formulate such strings/queries 40-44 and we do not test the completeness of the underlying dataset provided by the search system.”
BackgroundAppelbaum, M., Cooper, H., Kline, R. B., Mayo-Wilson, E., Nezu, A. M., & Rao, S. M. (2018) · American Psychologist · doi:10.1037/amp0000191 · Cited by 1,449
Self-citationCooper, C., Booth, A., Varley-Campbell, J., Britten, N., & Garside, R. (2018) · BMC Medical Research Methodology · doi:10.1186/s12874-018-0545-3 · Cited by 561
Searching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647
Introduction to PRISMA 2020 and implications for research synthesis methodologists
Page, M. J., Moher, D., & McKenzie, J. E. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1535 · Cited by 270
Metadata
- Authors
- Atkinson, K. M., Koenka, A. C., Sanchez, C. E., Moshontz, H., & Cooper, H.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2014-09-24
- Volume
- 6
- Issue
- 1
- Pages
- 87-95
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W1580073794
- Cited by
- 125
- References
- 9
- FWCI
- 3.84
- Type
- journal article
- Language
- en
- Open access
- green
- Funders
- WT Grant Foundation (181179)
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. https://app.cochat.ai/papers/10.1002/jrsm.1127
- The paper
- Atkinson, K. M., Koenka, A. C., Sanchez, C. E., Moshontz, H., & Cooper, H. (2015). Reporting standards for literature searches and report inclusion criteria: making research syntheses more transparent and easy to replicate. Research Synthesis Methods. https://doi.org/10.1002/jrsm.1127
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.