The value of a second reviewer for study selection in systematic reviews
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Background: Although dual independent review of search results by two reviewers is generally recommended for systematic reviews, there are not consistent recommendations regarding the timing of the use of the second reviewer. This study compared the use of a complete dual review approach, with two reviewers in both the title/abstract screening stage and the full‐text screening stage, as compared with a limited dual review approach, with two reviewers only in the full‐text stage. Methods: This study was performed within the context of a large systematic review. Two reviewers performed a complete dual review of 15 000 search results and a limited dual review of 15 000 search results. The number of relevant studies mistakenly excluded by highly experienced reviewers in the complete dual review was compared with the number mistakenly excluded during the full‐text stage of the limited dual review. Results: In the complete dual review approach, an additional 6.6% to 9.1% of eligible studies were identified during the title/abstract stage by using two reviewers, and an additional 6.6% to 11.9% of eligible studies were identified during the full‐text stage by using two reviewers. In the limited dual review approach, an additional 4.4% to 5.3% of eligible studies were identified with the use of two reviewers. Conclusions: Using a second reviewer throughout the entire study screening process can increase the number of relevant studies identified for use in a systematic review. Systematic review performers should consider using a complete dual review process to ensure all relevant studies are included in their review.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic reviews: CRD's guidance for undertaking reviews in health care
Tacconelli, E. (2010) · The Lancet Infectious Diseases · doi:10.1016/s1473-3099(10)70065-7 · Cited by 3,245
Finding What Works in Health Care: Standards for Systematic Reviews
Eden, J., Levit, L. A., Berg, A. O., & Morton, S. C. (2013) · Cited by 943
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“Novel methods for performing study selection in systematic reviews, such as automating the process,(8-12) have been proposed and have shown promise in providing a reduction in work load but also a reduction in precision.(6,13) These methods need further comparison in precision to the complete dual review process.”
MethodO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., & Wentz, R. (2002) · Statistics in Medicine · doi:10.1002/sim.1190 · Cited by 340
Finding What Works in Health Care: Standards for Systematic Reviews
Committee on Standards for Systematic Reviews of Comparative Effectiveness Research, Board on Health Care Services, & Institute of Medicine (2011) · doi:10.17226/13059 · Cited by 289
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
Shemilt, I., Khan, N., Park, S., & Thomas, J. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0315-4 · Cited by 168
Prioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan
Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1237 · Cited by 102
Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A. (2012) · Genetics in Medicine · doi:10.1038/gim.2012.7 · Cited by 62
Multimorbidity in randomized controlled trials of behavioral interventions: A systematic review.
Stoll, C. R. T., Izadi, S. R., Fowler, S., Philpott-Streiff, S. E., Green, P. A., Suls, J. M., Winter, A. C., & Colditz, G. A. (2019) · Health Psychology · doi:10.1037/hea0000726 · Cited by 38
Self-citationEPC Methods: An Exploration of the Use of Text-Mining Software in Systematic Reviews
Paynter, R. A., Bañez, L. L., Berliner, E., Erinoff, E., Lege‐Matsuura, J., Potter, S., & Uhl, S. (2016) · Europe PMC (PubMed Central) · Cited by 31
Commentary on EPC methods: an exploration of the use of text-mining software in systematic reviews
Paynter, R., Bañez, L. L., Erinoff, E., Lege-Matsuura, J., & Potter, S. (2017) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.11.019 · Cited by 17
Cited by
Gartlehner, G., Affengruber, L., Titscher, V., Noel-Storr, A., Dooley, G., Ballarini, N., & König, F. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.005 · Cited by 206
Transforming literature screening: The emerging role of large language models in systematic reviews
Delgado-Chaves, F. M., Jennings, M. J., Atalaia, A., Wolff, J., Horvath, R., Mamdouh, Z. M., Baumbach, J., & Baumbach, L. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2411962122 · Cited by 86
Moe-Byrne, T., Shepherd, J., Merecz-Kot, D., Sinokki, M., Naumanen, P., Hakkaart-van Roijen, L., & Van Der Feltz-Cornelis, C. (2022) · PLOS Digital Health · doi:10.1371/journal.pdig.0000123 · Cited by 55
“Firstly, we only searched google scholar for grey literatures and there was a possibility that some studies could have been missed due to double screening for only 10% of title and abstract [67,68].”
MethodHan, A., & Kim, T. H. (2022) · Journal of Psychiatric Research · doi:10.1016/j.jpsychires.2022.05.031 · Cited by 26
Huggins, L. K. L., Min, S. H., Kaplan, S., Wei, J., Welsh-Bohmer, K., & Xu, H. (2023) · Journal of Alzheimer's Disease · doi:10.3233/jad-221167 · Cited by 18
Metadata
- Authors
- Stoll, C. R. T., Izadi, S., Fowler, S., Green, P., Suls, J., & Colditz, G. A.
- Venue
- Research Synthesis Methods
- Publisher
- Wiley
- Published
- 2019-07-18
- Volume
- 10
- Issue
- 4
- Pages
- 539-545
- ISSN
- 1759-2879 (ISSN-L)
- 1759-2887
- OpenAlex
- W2953730938
- Cited by
- 297
- References
- 14
- FWCI
- 3.75
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.