Error rates of human reviewers during abstract screening in systematic reviews
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references. OBJECTIVES: To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches. METHODS: We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies. RESULTS: We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%). CONCLUSIONS: This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is important to note that humans typically miss around 10% of the relevant records [52]”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since most SLRs ultimately exclude a substantial number of potentially eligible articles (Sampson et al., 2011; Wang et al., 2020; Yaffe et al., 2012), the use of highly sensitive LLM tools to rapidly screen out obviously irrelevant articles would in itself substantially reduce researchers’ workloads while ensuring that almost all relevant articles are retained for detailed manual review.”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Wang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the worst possible documented prediction, human error rates reached up to 40% (Wang et al., 2020).”
From Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Wang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The process of systematic reviewing is error prone and extremely time intensive 10 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [10]
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“A recent study found an automation tool used by health systems to identify patients with complex health needs led to significant racial bias.[23] Assessment of these automation tools for systematic reviews, thus, is critical for wide adoption in practice.”
MethodCochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 1×
“It is estimated that the annual publications of systematic reviews increased 2,728% from 1,024 in 1991 to 28,959 in 2014.[2] Despite of the surging number of published systematic reviews in recent years, many systematic reviews employ suboptimal methodological approaches.”
BackgroundEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 1×
“A rigorous systematic review follows strict approaches and requires significant resource and time to complete, which typically lasts 6–18 months by a team of human reviewers.[25] Automation tools have the potential to mimic human activities in systematic review tasks and Fig 1.”
BackgroundExpediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“[7, 8, 14–21] Automation tools for systematic reviews, based on machine learning, text mining, and natural language processing, have particularly been popular with an estimated workload reduction from 30% to 70%.[14] Till July 2019, 39 tools have been completed and are available for “real-world” use.”
BackgroundMurad, M. H., Montori, V. M., Ioannidis, J. P. A., Jaeschke, R., Devereaux, P. J., Prasad, K., Neumann, I., Carrasco-Labra, A., Agoritsas, T., Hatala, R., et al. (2014) · JAMA · doi:10.1001/jama.2014.5559 · Cited by 454 · cited 1×
“[2–4] Rigorous systematic reviews require strict procedures with at least eight time-consuming steps.[5, 6] Significant time and resources are needed, with estimated 0.”
BackgroundSelf-citationReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Research works printed but not linked: 2
- Cochrane AL. 1931–1971: a critical review, with particular reference to the medical profession. Medicines for the year. 2000; 1979: 1.
- Wang Z, Noor A, Elraiyah T, Murad M, editors. Dual monitors to increase efficiency of conducting systematic reviews. 21st Cochrane Colloquium; 2013. 25085736
Cited by
Page, M. J., Moher, D., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., Shamseer, L., Tetzlaff, J. M., Akl, E. A., Brennan, S. E., et al. (2021) · BMJ · doi:10.1136/bmj.n160 · Cited by 13,262
Self-citationAn open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
Page, M. J., Moher, D., Bossuyt, P., Boutron, I., Hoffmann, T., mulrow, C., Shamseer, L., Tetzlaff, J., Akl, E., Brennan, S. E., et al. (2020) · doi:10.31222/osf.io/gwdhk · Cited by 916
Self-citationArtificial intelligence and the conduct of literature reviews
Wagner, G., Lukyanenko, R., & Paré, G. (2022) · Journal of Information Technology · doi:10.1177/02683962211048201 · Cited by 296
“In fact, recent evidence in the health sciences suggests a base rate of 10% disagreements between inclusion screens conducted independently (Wang et al., 2020).”
Khraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2023) · arXiv · doi:10.48550/arxiv.2310.17526 · Cited by 268
“27 This suggests that accuracy scores between 60% and 80% could be regarded as “ near-par, ” and scores below 60% could be considered “ subpar.”
Artificial intelligence in systematic reviews: promising when appropriately used
van Dijk, S. H. B., Brusse-Keizer, M. G. J., Bucsán, C. C., van der Palen, J., Doggen, C. J. M., & Lenferink, A. (2023) · BMJ Open · doi:10.1136/bmjopen-2023-072254 · Cited by 256
“32 Second, the AI tool may not have properly ranked highly relevant to irrelevant articles.”
BackgroundKhraisha, Q., Put, S., Kappenberg, J., Warraitch, A., & Hadfield, K. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1715 · Cited by 269
“27 This suggests that accuracy scores between 60% and 80% could be regarded as “ near-par, ” and scores below 60% could be considered “ subpar.”
Understanding urban perception with visual data: A systematic review
Ito, K., Kang, Y., Zhang, Y., Zhang, F., & Biljecki, F. (2024) · Cities · doi:10.1016/j.cities.2024.105169 · Cited by 207
“…activities, mental health (Buttazzoni & Minaker, 2022; Buttazzoni & Minaker, 2023; Luo & Jiang, 2022), and restorativeness (Zieff et al., 2018; Vera-Villarroel et al., 2016; Zhao, Wu, & Wang, 2020), indicating a shift towards integrating health and aesthetic considerations in street design.”
Influential(2023) · doi:10.21203/rs.3.rs-2856011/v1 · Cited by 145
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140
“, incorrectly included or incorrectly excluded at title/ abstract screening) among 139,467 citations that underwent 329,332 inclusion and exclusion decisions [3].”
BackgroundBoetje, J., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02502-7 · Cited by 146
“It is important to note that humans typically miss around 10% of the relevant records [52], and some relevant records may not be included in the dataset due to limitations in the search or errors in the metadata of records.”
BackgroundEvaluating the effectiveness of large language models in abstract screening: a comparative analysis
Li, M., Sun, J., & Tan, X. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02609-x · Cited by 125
“While the depth of human expertise brought about nuanced understanding, the manual nature of this method made it both time-consuming and, at times, prone to human error [6, 9].”
BackgroundA synthesis of evidence for policy from behavioural science during COVID-19
Ruggeri, K., Stock, F., Haslam, S. A., Capraro, V., Boggio, P. S., Ellemers, N., Cichocka, A. K., Douglas, K. M., Rand, D. G., van der Linden, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06840-9 · Cited by 144
“Conducting a systematic review would have required refining each of the claims to be more specific than originally written, which risks excluding a large amount of potentially relevant evidence 137 .”
BackgroundAn open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Graafsma, J., Murphy, R. M., van de Garde, E. M. W., Karapinar‐Çarkit, F., Derijks, H. J., Hoge, R. H. L., Klopotowska, J. E., & van den Bemt, P. M. L. A. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae076 · Cited by 135
“32,33 The algorithm does not choose the included studies but merely presents the studies to the researchers in order of probability of relevance based on prior knowledge.”
BackgroundPerformance of a Large Language Model in Screening Citations
Oami, T., Okada, Y., & Nakada, T.-A. (2024) · JAMA Network Open · doi:10.1001/jamanetworkopen.2024.20496 · Cited by 84
Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2023) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-023-01450-0 · Cited by 94
“Additionally, using the machine learning classifier displayed articles that are more relevant first, potentially decreasing the error rate in traditional reviews [121].”
MethodLi, Y., Datta, S., Rastegar-Mojarad, M., Lee, K., Paek, H., Glasgow, J., Liston, C., He, L., Wang, X., & Xu, Y. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf030 · Cited by 69
“10 Systematic literature reviews often retrieve thousands of citations through initial keyword dataset searches.”
Tran, V.-T., Gartlehner, G., Yaacoub, S., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Alebouyeh, F., Afach, S., Meerpohl, J., et al. (2024) · Annals of Internal Medicine · doi:10.7326/m23-3389 · Cited by 66
“Reappraisal of discrepancies between humans and GPT identi fi ed 63 citations (median, 6 per review [IQR, 4 to 7]), across all reviews, identi fi ed by GPT models but wrongly excluded by humans at the title and abstract level, of which 7 were actually eligible at the full-text level ( Supplement…”
BackgroundAgnello, D. M., Loisel, Q. E. A., An, Q., Balaskas, G., Chrifou, R., Dall, P., de Boer, J., Delfmann, L. R., Giné‐Garriga, M., Goh, K., et al. (2023) · Journal of Medical Internet Research · doi:10.2196/45059 · Cited by 57
“Our false-negative rate was 9% (6/64), which is the average error rate lower than what is achievable by a human alone, which is about 10% on average [33].”
Kebede, M. M., Le Cornet, C., & Fortner, R. T. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1589 · Cited by 50
“The recent study concluded the total error rate (false inclusion/exclusion) in manual title/abstract screening is about 11%.(49) Adjustment for human screening error might have yielded better performance as documented in the recent studies(13,49).”
BackgroundMatsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2024) · Journal of Medical Internet Research · doi:10.2196/52758 · Cited by 50
“In addition, human error is inevitable in the screening process [15-17], and the number of such errors can increase as the amount of paper to be screened increases possibly due to fatigue and cognitive overload [18,19].”
Tóth, B., Berek, L., Gulácsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 48
Alhumaid, S., Mutair, A. S. A., Alawi, Z. A., Alhmeed, N., Zaidi, A. R. Z., & Tobaiqy, M. A. (2020) · Tropical Medicine and Infectious Disease · doi:10.3390/tropicalmed5040180 · Cited by 50
“Disagreements between the two reviewers after full text screening were reconciled via consensus by a third reviewer (AA) [22].”
BackgroundHow Stimulating Is a Green Stimulus? The Economic Attributes of Green Fiscal Spending
O’Callaghan, B., Yau, N., & Hepburn, C. J. (2022) · Annual Review of Environment and Resources · doi:10.1146/annurev-environ-112420-020640 · Cited by 65
“Automated text analysis can help to address these issues (79, 81, 83, 84); however, such analyses also face several practical and technical constraints (84).”
BackgroundStogiannos, N., Carlier, S., Harvey-Lloyd, J. M., Brammer, A., Nugent, B., Cleaver, K., McNulty, J. P., dos Reis, C. S., & Malamateniou, C. (2021) · Autism · doi:10.1177/13623613211065542 · Cited by 53
“Our study relied on two separate reviewers and followed all relevant guidelines, but it is possible that some studies were missed (Wang et al., 2020).”
MethodAutomatic classification of literature in systematic reviews on food safety using machine learning
van den Bulk, L. M., Bouzembrak, Y., Gavai, A., Liu, N., van den Heuvel, L. J., & Marvin, H. J. P. (2022) · Current Research in Food Science · doi:10.1016/j.crfs.2021.12.010 · Cited by 38
“Furthermore, since the class labels of the data were set by human reviewers, who can make mistakes in their labelling during systematic reviews (Wang et al., 2020), it is better to allow a bit of room in the recall of the model.”
BackgroundHamel, C., Michaud, A., Thuku, M., Affengruber, L., Skidmore, B., Nussbaumer-Streit, B., Stevens, A., & Garritty, C. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.06.027 · Cited by 36
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., et al. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1178181 · Cited by 36
“Note that humans typically fail to find about 10% due to screening fatigue (Wang et al., 2020).”
BackgroundGalli, C., Gavrilova, A. V., & Calciolari, E. (2025) · Information · doi:10.3390/info16050378 · Cited by 36
Joy-Camacho, W., & Thornhill, I. (2024) · Journal of Environmental Management · doi:10.1016/j.jenvman.2024.121749 · Cited by 56
Böke, H., Aygün, Y., Tüfekçi, Ş., Yağın, F. H., Canpolat, B., Norman, G., Prieto-González, P., & Ardigò, L. P. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1508808 · Cited by 163
“Subsequently, two authors (H.B. and P.P.-G.) conducted independent assessments of the full-text studies for inclusion (Waffenschmidt et al., 2019; Wang et al., 2020).”
Vicary, E., Kapadia, D., Bee, P., Bennion, M., & Brooks, H. L. (2024) · Journal of Mental Health · doi:10.1080/09638237.2024.2408237 · Cited by 69
“The error rate during the screening process, 0.27% at title and abstract and 1.42% at full-text, never exceeded the recommended 10.76% (Wang et al., 2020) and so double screening all potential studies was not considered necessary.”
InfluentialBrück, C. C., Mooldijk, S. S., Kuiper, L. M., Sambou, M. L., Licher, S., Mattace-Raso, F. U. S., & Wolters, F. J. (2025) · BMJ · doi:10.1136/bmj-2024-080636 · Cited by 60
König, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1762 · Cited by 40
“17 While fatigue can also affect the quality of any AI-AS, the smaller number of items requiring screening could reduce this risk.”
InfluentialKönig, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2023) · doi:10.31234/osf.io/ybu3w · Cited by 33
Westendorp, J., Geerse, O. P., van der Lee, M. L., Schoones, J. W., van Vliet, M. H. M., de Wit, T. F. R., Evers, A. W. M., & van Vliet, L. M. (2023) · Psycho-Oncology · doi:10.1002/pon.6247 · Cited by 33
“36 To optimize the reliability of using ASReview we well prepared the AI screening process by means of the 20% double screening.”
MethodPerlman‐Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1636 · Cited by 30
“To evaluate precision and recall, we first calculated the baseline expected variability due to human error in screening, by comparing the included and excluded pilot abstracts between the pre-project and without-tool stages, as there is inherent human error in the systematic review process.(31) We then used Fisher's e…”
MethodAccuracy of Large Language Models for Literature Screening in Thoracic Surgery: Diagnostic Study
Dai, Z.-Y., Wang, F., Shen, C., Ji, Y.-L., Li, Z.-Y., Wang, Y., & Pu, Q. (2025) · Journal of Medical Internet Research · doi:10.2196/67488 · Cited by 45
Choosing the Right Tool for the Job: Screening Tools for Systematic Reviews in Education
Zhang, Q., & Neitzel, A. (2024) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2023.2209079 · Cited by 28
“For human reviewers, an empirical study found 10.76% total error rate for false inclusion and exclusion (Wang et al., 2020).”
InfluentialCroon, P. M., Selder, J. L., Allaart, C. P., Bleijendaal, H., Chamuleau, S. A. J., Hofstra, L., Išgum, I., Ziesemer, K. A., & Winter, M. M. (2022) · European Heart Journal - Digital Health · doi:10.1093/ehjdh/ztac035 · Cited by 41
“Moreover, one by one screening by humans is far from perfect leading to 10% missed articles.(52) To reduce the chance of relevant missed articles, two separate reviewers conducted the full selection protocol which did not lead to largely different results.”
BackgroundMcDonald, S. J., Sharp, S., Morgan, R. L., Murad, M. H., & Navarro, D. F. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.12.023 · Cited by 31
Self-citationAbogunrin, S., Muir, J. M., Zerbini, C., & Sarri, G. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1454245 · Cited by 27
Systematic review and meta-analysis of preclinical studies
Ineichen, B. V., Held, U., Salanti, G., Macleod, M. R., & Wever, K. E. (2024) · Nature Reviews Methods Primers · doi:10.1038/s43586-024-00347-x · Cited by 32
“Despite ARRIVE guidelines aiming to enhance reporting standards 113 , many studies fail to disclose critical details such as the number of animals used or whether means or medians are reported.”
Tercero-Hidalgo, J. R., Khan, K. S., Bueno‐Cavanillas, A., López, R. F., Huete, J. F., Amezcua‐Prieto, C., Zamora, J., & Fernández-Luna, J. M. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.04.027 · Cited by 24
“On average, it takes 15months for teams of five reviewers to complete a traditional systematic review [51], with estimated screening error rates of around 10% [52].”
Liu, Y. C., Zhao, S., Zhang, X. R., Zhang, X., Liang, T., & Ning, Z. (2025) · Behavioral Sciences · doi:10.3390/bs15050685 · Cited by 48
“This method diminishes the quantity of abstracts necessitating evaluation by the human reviewer to encompass all eligible papers, hence enhancing the efficiency of the selection process (Holzinger, 2016; Wang et al., 2020).”
Assessing the Integrity of Clinical Trials Included in Evidence Syntheses
Núñez‐Núñez, M., Ibáñez, N. C., Zamora, J., Bueno‐Cavanillas, A., & Khan, K. S. (2023) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph20126138 · Cited by 26
“, the selection process suffers a 10% error (false inclusion and false exclusion) rate [50].”
BackgroundLu, C., Wu, S., Ke, L., Liu, F., Shang, W., Deng, X., Huang, Y., Zhang, Q., Cui, X., Mentis, A. A., et al. (2022) · Frontiers in Pharmacology · doi:10.3389/fphar.2022.901875 · Cited by 24
Self-citationMachine learning to optimize literature screening in medical guideline development
Harmsen, W., de Groot, J., Harkema, A., van Dusseldorp, I., de Bruin, J., van den Brand, S., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02590-5 · Cited by 21
“Note that humans typically misclassify about 10% [34].”
BackgroundWongsa, C., Phinyo, P., Sompornrattanaphan, M., Krikeerati, T., Lumkul, L., & Thongngarm, T. (2022) · The Journal of Allergy and Clinical Immunology In Practice · doi:10.1016/j.jaip.2022.01.046 · Cited by 24
An update on tests used for intraoperative monitoring of cognition during awake craniotomy
de Zwart, B., & Ruis, C. (2024) · Acta Neurochirurgica · doi:10.1007/s00701-024-06062-6 · Cited by 26
“That being said, screening by humans remains imperfect and mistakes can have been made during the labeling of studies [283].”
BackgroundResidential energy efficiency interventions: A meta-analysis of effectiveness studies
Berretta, M., Furgeson, J. R., Wu, N., Zamawe, C. O. F., Hamilton, I., & Eyers, J. E. (2021) · Campbell Systematic Reviews · doi:10.1002/cl2.1206 · Cited by 30
“SRs often exclude the vast majority of studies identified through comprehensive searches (Wang et al., 2020).”
BackgroundGans, E. A., van Mun, L., de Groot, J. F., Munster, B. C. V., Rake, E. A., van Weert, J. C. M., Festen, S., & van den Bos, F. (2023) · Patient Education and Counseling · doi:10.1016/j.pec.2023.107981 · Cited by 20
Bourke, M., Wang, H., Fortnum, K., Thomas, G., O’Flaherty, M., Mulcahy, S. K., Gomersall, S. R., Alsop, T., Trost, S. G., Bruijns, B. A., et al. (2025) · Scandinavian Journal of Medicine and Science in Sports · doi:10.1111/sms.70120 · Cited by 25
Scotti, K. L., Young, S. M. N., Gainey, M. A., & Lan, H. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70046 · Cited by 37
“In general, a benchmark for ML ‐ assisted screening is to maintain recall rates of at least 95% to ensure that the process performs at a level comparable to human reviewers [53, 185].”
Reproducibility and Data Storage for Active Learning-Aided Systematic Reviews
Lombaers, P., de Bruin, J., & van de Schoot, R. (2024) · Applied Sciences · doi:10.3390/app14093842 · Cited by 20
“Humans make mistakes when reading and labeling records [7], and making one different labeling decision will result in a different set of records shown to the annotator.”
A Framework for Evidence Synthesis Programs to Respond to a Pandemic
Murad, M. H., Nayfeh, T., Urtecho, M., Seisa, M. O., Abd‐Rabu, R., Farah, M. H., Firwana, M., Hasan, B., Jawaid, T., Shah, S. H., et al. (2020) · Mayo Clinic Proceedings · doi:10.1016/j.mayocp.2020.05.009 · Cited by 18
“For example, not duplicating the process of study selection by two independent reviewers can introduce an error rate of approximately 11%.(6) Within weeks of the pandemic, multiple systematic reviews had been published about the same topic, which led to a challenge of determining which review to use, which one is most…”
BackgroundSelf-citationBleijendaal, H., Croon, P. M., Pool, M. D. O., Malekzadeh, A., Aufiero, S., Amin, A. S., Zwinderman, A. H., Pinto, Y. M., Wilde, A. A. M., & Winter, M. M. (2023) · Trends in Cardiovascular Medicine · doi:10.1016/j.tcm.2022.01.011 · Cited by 17
Perceived physical environment and active transport in adolescents: A systematic review
Klos, L., Eberhardt, T., Nigg, C., Niessner, C., Wäsche, H., & Wöll, A. (2023) · Journal of Transport & Health · doi:10.1016/j.jth.2023.101689 · Cited by 25
Oami, T., Okada, Y., Sakuraya, M., Fukuda, T., Shime, N., & Nakada, T. (2023) · Journal of Epidemiology · doi:10.2188/jea.je20230227 · Cited by 19
Optimal large language models to screen citations for systematic reviews
Oami, T., Okada, Y., & Nakada, T. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10014 · Cited by 29
“5,6,16 As the performance of LLMs continues to evolve, using the latest models or integrating multiple LLMs may represent promising strategies for enhancing their accuracy and efficiency related to citation screening.”
Rashid, M., Yi, C. S., Sathapanasiri, T., Udayachalerm, S., Boonpattharatthiti, K., Insuk, S., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., Dhippayom, T., et al. (2025) · Value in Health · doi:10.1016/j.jval.2025.07.001 · Cited by 26
Hilkenmeier, F., Pelzer, M., Stierle, C. M. G., & Fink‐Lamotte, J. (2025) · Social Science Computer Review · doi:10.1177/08944393251404052 · Cited by 23
ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews.Preprint
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2020) · arXiv (Cornell University) · Cited by 21
Hester, K., Kirrane, E. F., Anderson, T., Kulikowski, N., Simmons, J. E., & Lehmann, D. M. (2022) · Environment International · doi:10.1016/j.envint.2022.107528 · Cited by 15
Teijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Siso, P. V., van den Brand, S. A. G. E., Bockting, C., van de Schoot, R., et al. (2022) · doi:10.31234/osf.io/t7bpd · Cited by 14
Lu, T., Kong, B., Wang, Y., Yu, J., Pan, Y., Chen, D., Li, H., Chen, X., Yuan, Z., Yang, Z., et al. (2023) · Journal of Ethnopharmacology · doi:10.1016/j.jep.2023.117267 · Cited by 15
Thomas, I. N., Roche, P. K., & Grêt‐Regamey, A. (2024) · Ecological Informatics · doi:10.1016/j.ecoinf.2024.102819 · Cited by 18
Validation and usability study of the framework for a user needs-centered mHealth app selection
Lee, J., & Lee, M. A. (2022) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2022.104877 · Cited by 19
Vaseur, R. M. E., Braake, E. T., Beinema, T. C., Nijeweme-d’Hollosy, W. O., & Tabak, M. (2024) · Patient Education and Counseling · doi:10.1016/j.pec.2024.108267 · Cited by 18
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · BMC Public Health · doi:10.1186/s12889-024-18907-x · Cited by 34
“Additionally, the extremely imbalanced nature of data found in database searches (i.e., significantly more irrelevant than relevant titles and abstracts) means that traditional reviews may be error prone [94].”
Almukhtar, A., Batcup, C., Bowman, M., Beatty, J. W., Leff, D. R., Demirel, P., Judah, G., & Porat, T. (2024) · International Journal of Surgery · doi:10.1097/js9.0000000000001951 · Cited by 15
“Consequently, it yielded an extensive number of irrelevant articles increasing the chances of human error [115] .”
Phillips, S. M., Clevenger, K. A., Bruijns, B. A., Tucker, P., Vanderloo, L. M., Loh, A., Naveed, M., & Bourke, M. (2024) · Journal for the Measurement of Physical Behaviour · doi:10.1123/jmpb.2023-0060 · Cited by 18
“The use of machine learning to assist with systematic reviews has been shown to reduce reviewer error in the screening process (Wang et al., 2020).”
GPT-3.5 Turbo and GPT-4 Turbo in Title and Abstract Screening for Systematic Reviews
Oami, T., Okada, Y., & Nakada, T. (2025) · JMIR Medical Informatics · doi:10.2196/64682 · Cited by 13
“Manual citation screening, however, is a time-consuming and labor-intensive process, often resulting in human errors and increased workloads [1,2].”
Sharma, B. M., Kalina, J., Whaley, P., & Scheringer, M. (2021) · Environment International · doi:10.1016/j.envint.2021.106437 · Cited by 14
Tran, V.-T., Gartlehner, G., Yaacoub, S. G., Boutron, I., Schwingshackl, L., Stadelmaier, J., Sommer, I., Aboulayeh, F., Afach, S., Meerpohl, J. J., et al. (2023) · medRxiv · doi:10.1101/2023.12.15.23300018 · Cited by 13
Do it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13
Transformative Automation: AI in Scientific Literature Reviews
Zala, K., Acharya, B., Mashru, M., Palaniappan, D., Gerogiannis, V. C., Kanavos, A., & Karamitsos, I. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01501122 · Cited by 21
“When considering the potential of AI assistance for these steps, it’s crucial to recognize that the reliability of manual screening methods should not be overstated, as even screenings conducted by experts exhibit a disagreement rate of 10% on average [81].”
BackgroundSun, J., Caputo, V., & Taylor, H. (2024) · Applied Economic Perspectives and Policy · doi:10.1002/aepp.13446 · Cited by 18
Tanner, V., Millet, G. P., & Bourdillon, N. (2024) · Sports Medicine - Open · doi:10.1186/s40798-024-00768-8 · Cited by 19
“This tool’s false exclusion rate [62, 63] is comparable with human performance [64–66] and has thus been suggested as a second screener alternative [67–71].”
ResultZeraatkar, D., Ling, M., Kirsh, S., Jassal, T., Pitre, T., Chakraborty, S. P., Turner, T., Turkstra, L. S., McIntyre, R. S., Izcovich, A., et al. (2025) · BMJ Open · doi:10.1136/bmjopen-2024-086407 · Cited by 18
Reporting feedback on healthcare outcomes to improve quality in care: a scoping review
Ali, M. P., Visser, E. H., West, R. L., van Noord, D., van der Woude, C. J., & van Deen, W. K. (2025) · Implementation Science · doi:10.1186/s13012-025-01424-9 · Cited by 14
“This combined approach balanced nuanced interpretation with rapid processing, consistency, and error reduction [41, 42].”
How Well Do ChatGPT and Claude Perform in Study Selection for Systematic Review in Obstetrics
Insuk, S., Boonpattharatthiti, K., Booncharoen, C., Chaipitak, P., Rashid, M., Veettil, S. K., Lai, N. M., Chaiyakunapruk, N., & Dhippayom, T. (2025) · Journal of Medical Systems · doi:10.1007/s10916-025-02246-4 · Cited by 15
“Wang et al. (2020) found an abstract screening error rate of about 10.8% by human reviewers (false inclusions or exclusions), with error rates varying between roughly 5% and 21% depending on the topic area [3].”
InfluentialKallmes, K. M., Thurnham, J., Sauca, M., Tarchand, R., Kallmes, K. R., & Holub, K. J. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70059 · Cited by 17
“Smart Search had 1.81% Precision; the expected Precision in expert ‐ drafted searches ranges from 1% to 5%, with the largest study to date by Wang et al. finding Precision of 5.3% [22].”
InfluentialRetractedBrennan, C., Streight, E., Cheng, S., & Rhodes, R. E. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01778-9 · Cited by 26
“Locating all relevant records is extremely challenging, even for humans who typically overlook approximately 10% of relevant records [39].”
Chan, G. C. K., He, E., Leung, J., & Verspoor, K. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.1 · Cited by 10
Scotti, K. L., Jiao, C., Gainey, M. A., Bongiovanni, E. A., & Slayton, E. R. (2025) · The Journal of Academic Librarianship · doi:10.1016/j.acalib.2025.103054 · Cited by 23
Zuo, C., Yang, X., Errickson, J., Li, J., Hong, Y., & Wang, R. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00358-5 · Cited by 17
“However, manual screening is not only time-consuming and labor-intensive but also prone to human error [16], especially when an SR involves a large volume of diverse and context-specific studies.”
InfluentialBoetje, J., & van de Schoot, R. (2023) · doi:10.31234/osf.io/c93gq · Cited by 10
Teo, L., Elswyk, M. E. V., Lau, C. S., & Shanahan, C. J. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02374-3 · Cited by 13
“…where one of the axes indicates a study’s eligibility qualification status (e.g., whether a study should have been included/excluded) and the other axis indicates the study’s actual eligibility status (e.g., whether a study has actually been included/excluded) as determined by the reviewer [12].”
BackgroundHuber, N., Meester, M., Sassu, E. L., Waller, E. S. L., Krumova-Valcheva, G. L., Aprea, G., D’Angelantonio, D., Zoche-Golob, V., Scattolini, S., Marriott, E., et al. (2024) · Frontiers in Veterinary Science · doi:10.3389/fvets.2024.1494870 · Cited by 13
The hunt for the last relevant paper: blending the best of humans and AI
van de Schoot, R., Messina Coimbra, B., Evenhuis, T., Lombaers, P., Weijdema, F., de Bruin, L., Neeleman, R., Grandfield, E., Sijbrandij, M., Teijema, J. J., et al. (2025) · European Journal of Psychotraumatology · doi:10.1080/20008066.2025.2546214 · Cited by 6
“Moreover, screening the identified literature is extremely labour-intensive, and humans are prone to errors in such repetitive tasks (Wang et al., 2020) – a domain where AI excels, but is not necessarily flawless, as we demonstrate in the current paper.”
Large language model-generated clinical practice guideline for appendicitis
Boyle, A., Huo, B., Sylla, P. A., Calabrese, E., Kumar, S. S., Slater, B. J., Walsh, D. S., & Vosburg, R. W. (2025) · Surgical Endoscopy · doi:10.1007/s00464-025-11723-3 · Cited by 12
“has a degree of inconsistency and error [8], while LLMs may allow for a less biased and efficient approach.”
Why can’t epidemiology be automated (yet)?
Bann, D., Lowther, E., Wright, L., & Kovalchuk, Y. (2026) · International Journal of Epidemiology · doi:10.1093/ije/dyaf210 · Cited by 9
“At face value, this breaks the software engineering principle to “automate repetitive tasks,” but an additional motivation for automation is to reduce errors: humans do not screen without error [13].”
Screen Time Among and Youth Children With Disabilities: A Systematic Review and Meta‐Analysis
Vanderloo, L. M., Bourke, M., Taylor, L. G., Phillips, S. M., Loh, A., Disimino, K., Bassett-Gunter, R. L., Koo, T., Thompson‐Hill, M., & Tucker, P. (2025) · Child Care Health and Development · doi:10.1111/cch.70136 · Cited by 15
“Additionally, using the machine learning classifier displayed articles that are more relevant first, potentially decreasing the error rate observed in traditional reviews (Wang et al. 2020).”
Soong, Y. F., Hollman, H., & Rhodes, R. E. (2025) · International Journal of Behavioral Nutrition and Physical Activity · doi:10.1186/s12966-025-01782-z · Cited by 10
“This allows for a more efficient screening, both in terms of reduced time, but also because human screening can be prone to error [50, 51].”
Cole, C. H., Hajikhani, A., Hylkilä, E., Paronen, E., & Pihkola, H. (2025) · The International Journal of Life Cycle Assessment · doi:10.1007/s11367-025-02508-w · Cited by 15
“However, reliance on human judgement and human annotations as a gold standard or ground truth has also been criticized (Wang et al. 2020).”
Matsui, K., Utsumi, T., Aoki, Y., Maruki, T., Takeshima, M., & Takaesu, Y. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4520426 · Cited by 7
Cassell, K., Ologunowa, A., Rastegar-Mojarad, M., Chun, B., Huang, Y.-L., Wang, D., & Cossrow, N. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1662202 · Cited by 15
“These time-and labor-intensive processes can also introduce reviewer errors (Clark et al., 2021; Wang et al., 2020), as well as long gaps between SLR initiation and final publication (Beller et al., 2013; Borah et al., 2017).”
Addison, S. M., Campion, E. D., Landsman, M. J., Rogalin, C. L., & Veeh, C. A. (2025) · Research on Social Work Practice · doi:10.1177/10497315251357190 · Cited by 7
“Wang et al. (2020) found that human screeners have an error rate greater than 10%, suggesting that the title and abstract screening phase (for brevity, hereafter “ abstract screening ” ) is especially susceptible to unreliability among screeners and that automation tools may be particularly helpful…”
Influential
Metadata
- Authors
- Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H.
- Affiliations
- Mayo Clinic (US)
- Corresponding author
- Wang, Zhen
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-01-14
- Volume
- 15
- Issue
- 1
- Pages
- e0227742
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6959565
- OpenAlex
- W2999536597
- Semantic Scholar
- dd9e64171e0c3e867739daaf5c5023aed6e2e3ab
- Cited by
- 217
- References
- 24
- Citations by year
- 2026: 49 · 2025: 59 · 2024: 54 · 2023: 22 · 2022: 15 · 2021: 11 · 2020: 7
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- systematic reviews, abstract screening, false exclusion, gold standard, full-text screening
- MeSH terms
- Abstracting and Indexing, Systematic Reviews as Topic, Algorithms, Electronic Data Processing, Humans, Research Design, Databases, Bibliographic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
