State‐of‐the‐Evidence Reviews: Advantages and Challenges of Including Grey Literature
Abstract
Background: Increasingly, health policy decision‐makers and professionals are turning to research‐based evidence to support decisions about policy and practice. Systematic reviews are useful for gathering, summarizing, and synthesizing published and unpublished research about clearly defined interventions. State‐of‐the‐evidence reviews are broader than traditional systematic reviews and may include not only published and unpublished research, but also published and unpublished non‐research literature. Decisions about whether to include this “grey literature” in a review are challenging and lead to many questions about whether the advantages outweigh the challenges. Aims: The primary purpose of this article is to describe what constitutes grey literature, and methods to locate it and assess its quality. The secondary purpose is to discuss the core issues to consider when making decisions to include grey literature in a state‐of‐the‐evidence review. Methods: A recent state‐of‐the‐evidence review is used as an exemplar to present advantages and challenges related to including grey literature in a review. Results: Despite the challenges, in the exemplar, inclusion of grey literature was useful to validate the results of a research‐based literature search. Conclusion: Decisions about whether to include grey literature in a state‐of‐the‐evidence review are complex. A checklist to assist in decision‐making was created as a tool to assist the researcher in determining whether it is advantageous to include grey literature in a review. Worldviews on Evidence‐Based Nursing 2006; 3(2):55–61.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This may have led to an overestimation of the interventions' effectiveness due to publication bias.
“This may have also led to an overestimation of the interventions’ effectiveness due to publication bias [77, 78].”
From School-based healthy eating interventions for adolescents aged 10–19 years: an umbrella review (2024) · cites this paper as [78]
Cites
Realist review - a new method of systematic review designed for complex policy interventions
Pawson, R., Greenhalgh, T., Harvey, G., & Walshe, K. (2005) · Journal of Health Services Research & Policy · doi:10.1258/1355819054308530 · Cited by 3,190 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
Gray literature: An important resource in systematic reviews
Paez, A. (2017) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12266 · Cited by 1,459 · see citing works
Godin, K., Stapleton, J., Kirkpatrick, S. I., Hanning, R. M., & Leatherdale, S. T. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0125-0 · Cited by 959 · see citing works
Adams, R. J., Smart, P., & Huff, A. S. (2017) · International Journal of Management Reviews · doi:10.1111/ijmr.12102 · Cited by 995 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2019) · Information and Software Technology · doi:10.1016/j.infsof.2018.09.006 · Cited by 831 · see citing works
Adams, J., Hillier-Brown, F. C., Moore, H. J., Lake, A. A., Araujo-Soares, V., White, M., & Summerbell, C. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0337-y · Cited by 486 · see citing works
A scoping study of frameworks for adapting public health evidence-based interventions
Escoffery, C., Lebow-Skelley, E., Udelson, H., Böing, E. A., Wood, R., Fernandez, M. E., & Mullen, P. D. (2019) · Translational Behavioral Medicine · doi:10.1093/tbm/ibx067 · Cited by 307 · see citing works
Garousi, V., Felderer, M., & Mäntylä, M. V. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2916008 · Cited by 210 · see citing works
When and what to automate in software testing? A multi-vocal literature review
Garousi, V., & Mäntylä, M. V. (2016) · Information and Software Technology · doi:10.1016/j.infsof.2016.04.015 · Cited by 182 · see citing works
Pitfalls of Systematic Reviews and Meta-Analyses in Imaging Research
McInnes, M. D. F., & Bossuyt, P. M. M. (2015) · Radiology · doi:10.1148/radiol.2015142779 · Cited by 99 · see citing works
South, J., Bagnall, A.-M., Hulme, C., Woodall, J., Longo, R., Dixey, R., Kinsella, K., Raine, G., Vinall-Collier, K., & Wright, J. (2014) · Health Services and Delivery Research · doi:10.3310/hsdr02350 · Cited by 63 · see citing works
School-based healthy eating interventions for adolescents aged 10–19 years: an umbrella review
Samad, N., Bearne, L., Noor, F. M., Akter, F., & Parmar, D. (2024) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-024-01668-6 · Cited by 41 · see citing works
“This may have also led to an overestimation of the interventions’ effectiveness due to publication bias [77, 78].”
BackgroundPublished versus unpublished dissertations in psycho‐oncology intervention research
Moyer, A., Schneider, S., Knapp‐Oliver, S. K., & Sohl, S. J. (2010) · Psycho-Oncology · doi:10.1002/pon.1561 · Cited by 27 · see citing works
Navigating the DevOps landscape
Zhang, X., Zhao, P., & Jaskolka, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112331 · Cited by 21 · see citing works
Poole, J. L., Bradford, J. D., & Siegel, P. (2019) · American Journal of Occupational Therapy · doi:10.5014/ajot.2019.030619 · Cited by 16 · see citing works
- Authors
- Benzies, K. M., Premji, S., Hayden, K. A., & Serrett, K.
- Affiliations
- University of Calgary (CA)
- Venue
- Worldviews on Evidence-Based Nursing
- Publisher
- Wiley
- Published
- 2006-06
- Volume
- 3
- Issue
- 2
- Pages
- 55-61
- ISSN
- 1545-102X (ISSN-L)
- 1741-6787
- OpenAlex
- W2008155708
- Cited by
- 408
- References
- 1
- Citations by year
- 2026: 39 · 2025: 56 · 2024: 50 · 2023: 45 · 2022: 42 · 2021: 43 · 2020: 26 · 2019: 30 · 2018: 13 · 2017: 12 · 2016: 13 · 2015: 9 · 2014: 5 · 2013: 9 · 2012: 4
- FWCI
- 2.28
- Type
- journal article
- Language
- en
- Open access
- closed
- Locations
- Worldviews on Evidence-Based Nursing (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Canadian Health Services Research Foundation
- Alberta Centre for Child, Family and Community Research
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Business, Management and Accounting › Management Information Systems › Optics and Image Analysis
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Keywords
- grey literature, systematic reviews, literature searching, evidence-based health policy
- MeSH terms
- Decision Making, Female, Humans, Pregnancy, Review Literature as Topic, Nursing Research, Information Storage and Retrieval, Evidence-Based Medicine, Fetal Alcohol Spectrum Disorders
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
