Error rates of human reviewers during abstract screening in systematic reviews
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Automated approaches to improve the efficiency of systematic reviews are greatly needed. When testing any of these approaches, the criterion standard of comparison (gold standard) is usually human reviewers. Yet, human reviewers make errors in inclusion and exclusion of references. OBJECTIVES: To determine citation false inclusion and false exclusion rates during abstract screening by pairs of independent reviewers. These rates can help in designing, testing and implementing automated approaches. METHODS: We identified all systematic reviews conducted between 2010 and 2017 by an evidence-based practice center in the United States. Eligible reviews had to follow standard systematic review procedures with dual independent screening of abstracts and full texts, in which citation inclusion by one reviewer prompted automatic inclusion through the next level of screening. Disagreements between reviewers during full text screening were reconciled via consensus or arbitration by a third reviewer. A false inclusion or exclusion was defined as a decision made by a single reviewer that was inconsistent with the final included list of studies. RESULTS: We analyzed a total of 139,467 citations that underwent 329,332 inclusion and exclusion decisions from 86 unique reviewers. The final systematic reviews included 5.48% of the potential references identified through bibliographic database search (95% confidence interval (CI): 2.38% to 8.58%). After abstract screening, the total error rate (false inclusion and false exclusion) was 10.76% (95% CI: 7.43% to 14.09%). CONCLUSIONS: This study suggests important false inclusion and exclusion rates by human reviewers. When deciding the validity of a future automated study selection algorithm, it is important to keep in mind that the gold standard is not perfect and that achieving error rates similar to humans may be adequate and can save resources and time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“It is important to note that humans typically miss around 10% of the relevant records [52]”
From The SAFE procedure: a practical stopping heuristic for active learning-based screening in systematic reviews and meta-analyses (2024) · cites this paper as [52]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [3]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Since most SLRs ultimately exclude a substantial number of potentially eligible articles (Sampson et al., 2011; Wang et al., 2020; Yaffe et al., 2012), the use of highly sensitive LLM tools to rapidly screen out obviously irrelevant articles would in itself substantially reduce researchers’ workloads while ensuring that almost all relevant articles are retained for detailed manual review.”
From Analysis of article screening and data extraction performance by an AI systematic literature review platform (2025) · cites this paper as Wang et al., 2020
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In the worst possible documented prediction, human error rates reached up to 40% (Wang et al., 2020).”
From Can large language models replace humans in systematic reviews? Evaluating GPT ‐4's efficacy in screening and extracting data from peer‐reviewed and grey literature in multiple languages (2024) · cites this paper as Wang et al., 2020
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The process of systematic reviewing is error prone and extremely time intensive 10 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [10]
13 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Cochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1×
“A recent study found an automation tool used by health systems to identify patients with complex health needs led to significant racial bias.[23] Assessment of these automation tools for systematic reviews, thus, is critical for wide adoption in practice.”
MethodCochrane Handbook for Systematic Reviews of Interventions
Higgins, J. P. T., Green, S., & Ben Van Den, A. (2020) · International Coaching Psychology Review · doi:10.53841/bpsicpr.2020.15.2.123 · Cited by 5,670
The Mass Production of Redundant, Misleading, and Conflicted Systematic Reviews and Meta‐analyses
IOANNIDIS, J. P. A. (2016) · The Milbank Quarterly · doi:10.1111/1468-0009.12210 · Cited by 1,483 · cited 1×
“It is estimated that the annual publications of systematic reviews increased 2,728% from 1,024 in 1991 to 28,959 in 2014.[2] Despite of the surging number of published systematic reviews in recent years, many systematic reviews employ suboptimal methodological approaches.”
BackgroundEvidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232
Methodology in conducting a systematic review of systematic reviews of healthcare interventions
Smith, V., Devane, D., Begley, C. M., & Clarke, M. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-15 · Cited by 1,227 · cited 1×
“A rigorous systematic review follows strict approaches and requires significant resource and time to complete, which typically lasts 6–18 months by a team of human reviewers.[25] Automation tools have the potential to mimic human activities in systematic review tasks and Fig 1.”
BackgroundExpediting systematic reviews: methods and implications of rapid reviews
Ganann, R., Ciliska, D., & Thomas, H. (2010) · Implementation Science · doi:10.1186/1748-5908-5-56 · Cited by 752
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“[7, 8, 14–21] Automation tools for systematic reviews, based on machine learning, text mining, and natural language processing, have particularly been popular with an estimated workload reduction from 30% to 70%.[14] Till July 2019, 39 tools have been completed and are available for “real-world” use.”
BackgroundMurad, M. H., Montori, V. M., Ioannidis, J. P. A., Jaeschke, R., Devereaux, P. J., Prasad, K., Neumann, I., Carrasco-Labra, A., Agoritsas, T., Hatala, R., et al. (2014) · JAMA · doi:10.1001/jama.2014.5559 · Cited by 454 · cited 1×
“[2–4] Rigorous systematic reviews require strict procedures with at least eight time-consuming steps.[5, 6] Significant time and resources are needed, with estimated 0.”
BackgroundSelf-citationReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411
Research works printed but not linked: 2
- Cochrane AL. 1931–1971: a critical review, with particular reference to the medical profession. Medicines for the year. 2000; 1979: 1.
- Wang Z, Noor A, Elraiyah T, Murad M, editors. Dual monitors to increase efficiency of conducting systematic reviews. 21st Cochrane Colloquium; 2013. 25085736
Cited by
Items 101–200 of 217
Xu, S., Zhao, Z., Liu, X., & Meng, X.-L. (2025) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-025-03138-w · Cited by 8
“Recall reflects a tool’s ability to capture relevant studies, which is critical in systematic reviews to ensure that all relevant literature is included [35].”
Neurosynth Compose: A web-based platform for flexible and reproducible neuroimaging meta-analysis
Kent, J. D., Lee, N., Laird, A. R., Salo, T., Peraza, J., Bottenhorn, K. L., Oudyk, K., Nichols, T. E., Poline, J., & de la Vega, A. (2026) · Imaging Neuroscience · doi:10.1162/imag.a.1114 · Cited by 14
“This process is extraordinarily time consuming and error prone ( Wang et al., 2020): a single review can demand hundreds of researcher hours ( Borah et al., 2017).”
Enhancing recall in automated record screening: A resampling algorithm
Hou, Z., & Tipton, E. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1690 · Cited by 12
“As cited in Li et al, 3 the screening process can take up to 47 days 4 and $AUD 437,000 (about $USD 301,792 according to the average 2020 exchange rate).”
BackgroundReproducibility and Data storage Checklist for Active Learning-Aided Systematic ReviewsPreprint
Lombaers, P., Bruin, J. D., & van de Schoot, R. (2023) · doi:10.31234/osf.io/g93zf · Cited by 6
Zhang, X., Chang, Z., Zhao, S., & Ning, Z. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1662868 · Cited by 5
“ASReview predicts study relevance by training a classification model on labeled abstracts and continuously reprioritizes the remaining records according to their likelihood of inclusion (Holzinger, 2016; Van de Schoot et al., 2021; Wang et al., 2020).”
Zhang, X. R., Zhao, S., Ng, S., Liu, Y., Liang, T., Lao, C. I., & Ning, Z. (2025) · Frontiers in Psychology · doi:10.3389/fpsyg.2025.1618617 · Cited by 8
“This process enables ASReview to prioritize more relevant records for manual review, while reducing the overall screening burden (Holzinger, 2016; Wang et al., 2020).”
Ali, F., Tan, A. S.-C., & Wang, S. J.-W. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2024.16 · Cited by 9
“We chose 95% recall as humans generally can retrieve ~ 82% (based on abstract alone) or ~ 95% (based on full text) of relevant articles, 36 suggesting that 95% recall is as good as, if not better than, a human screener.”
König, L., Zitzmann, S., & Hecht, M. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10053 · Cited by 5
“The median SC was only about 1% lower when five relevant abstracts were included in the training set ( SC = 34.76% [19.46, 52.93]) compared to one relevant abstract (SC = 36.10% [20.43, 54.55]).”
InfluentialOhlsen, J. T., Søfteland, E., Akselsen, P. E., Aßmus, J., Harthug, S., Lein, T. E., Sevdalis, N., Wæhle, H. V., Øvretveit, J., & Hartveit, M. (2024) · BMJ Quality & Safety · doi:10.1136/bmjqs-2024-017185 · Cited by 7
“56 Further, we did not include purely qualitative studies on implementation factors if they did not report on implementation outcomes.”
Mental Qualities and Techniques in Tactical Populations: A Systematic Review
Raabe, J. J., Castillo, E. A., & Carl, J. (2021) · The Sport Psychologist · doi:10.1123/tsp.2020-0048 · Cited by 7
Lange, M. C., Li, R., Apolzan, J. W., Huber, P. R., Steliotes, E., Robertson, K., Wilson, N. L. W., Jain, K., Ramnath, R., Roe, B. E., et al. (2025) · Cleaner and Responsible Consumption · doi:10.1016/j.clrc.2025.100330 · Cited by 5
Hu, B., Tomini, E., Corrin, T., Pussegoda, K., Sandner, E., Henriques, A., Simniceanu, A., Fontana, L., Wagner, A., Brazeau, S., et al. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70042 · Cited by 7
“Two reviewer screening is a fundamental part of most evidence synthesis processes; however, research has also shown that an element of error remains with this approach and that humans may introduce bias into the process based on their experience and judgments [24].”
Predicted no effect concentrations of antifungals for wastewater management and agricultural use
Gil, D., José, S., Ascenso, A., Babič, M. N., Segal, E., Meletiadis, J., Gangneux, J. P., Weiskerger, C. J., Solo-Gabriele, H. M., Valério, E., et al. (2026) · Frontiers in Toxicology · doi:10.3389/ftox.2026.1767925 · Cited by 5
“Discrepancies were resolved by a third reviewer, following the approach recommended by Wang et al. (2020), which highlights potential error rates in manual abstract screening.”
Liu, Y., Zou, D., Dong, H.-W., Liu, Y. Y., Fu, E.-H., Tian, Z.-L., & Liu, N.-G. (2025) · PLoS ONE · doi:10.1371/journal.pone.0329349 · Cited by 4
Adam, T. J., Abosabie, S. A., Dittmer, M., Wolf, E., Abosabie, S. A., Behnke, C., Baier, F., Weickmann, A., Köser, L., Correll, C. U., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10093 · Cited by 12
Ritual Child Homicide in Contemporary Africa: A Systematic Review of the Empirical Literature
Owusu, E. S. (2024) · International Criminal Justice Review · doi:10.1177/10575677241282014 · Cited by 5
“Thus, even though single screening saves time and resources, there is a higher risk of missing relevant works/studies compared to double screening (Page et al., 2021b; Waffenschmidt et al., 2019; Wang et al., 2020).”
Leewis, S., Smit, K., van den Boom, B., & Versendaal, J. (2024) · Information and Software Technology · doi:10.1016/j.infsof.2024.107627 · Cited by 9
Nordmann, K., Schaller, M., Sauter, S., & Fischer, F. (2025) · JAMIA Open · doi:10.1093/jamiaopen/ooaf098 · Cited by 4
“47,48 Using the PCC scheme (Population-Concept-Context), frequently applied in scoping reviews, for providing inclusion criteria may be beneficial.”
Oami, T., Okada, Y., Maeda, K., & Nakada, T.-A. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1799623 · Cited by 4
Barak‐Corren, Y., Gupta, M., Tang, J. A., Smith, C. L., Callahan, R. M., Dori, Y., Rome, J. J., Gillespie, M. J., & O'Byrne, M. L. (2024) · Journal of the Society for Cardiovascular Angiography & Interventions · doi:10.1016/j.jscai.2024.102242 · Cited by 3
Zhang, Q., Pellegrini, M., Marsili, F., & Neitzel, A. J. (2024) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2024.1430355 · Cited by 3
Holland, C., Oakes, D. B., Sarna, M. K., Chai, K. E. K., Ng, L., & Moore, H. C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02511-7 · Cited by 3
“…RS reviewer team evaluated performance using the Work Saved over Sampling (WSS) metric [8] as used in other semi-auto-mated screening studies [4, 11], which calculates the percentage of documents a reviewer can avoid reading to identify 95% (WSS@95) and 100% (WSS@100) of the relevant articles…”
Courvoisier, D. S., Buitrago‐García, D., Buclin, C. P., Bürgisser, N., Iudici, M., & Mongin, D. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10054 · Cited by 6
“40,41 The second advantage is that requiring an agreement between a fixed number of LLMs before considering an answer as valid yielded accuracies surpassing both the human gold standard and the state-of-the-art LLMs.”
Leong, I. L., Wang, X., & Wu, H. (2026) · Neuroscience & Biobehavioral Reviews · doi:10.1016/j.neubiorev.2026.106566 · Cited by 5
Heijman, M. W. J., Hartog, M., van den Ende, C. H. M., Popa, C. D., & van den Bemt, B. J. F. (2026) · Drugs · doi:10.1007/s40265-025-02278-7 · Cited by 4
“…pump inhibitors, QoL quality of life, SGLT2 sodium-glucose cotransporter 2, TJR impact on time to total joint replacement *Including K&L, Kallman, GUSS, Altman, Croft and Verbruggen. comparison, human reviewers fail to identify about 10% of relevant papers when full datasets are screened [62].”
Oami, T., Okada, Y., & Nakada, T. (2026) · Hong Kong Journal of Emergency Medicine · doi:10.1002/hkj2.70085 · Cited by 3
Konzett, V., Laskou, F., Smolen, J. S., Edwards, C. J., Aletaha, D., van der Heijde, D. M. F. M., Winthrop, K. L., Takeuchi, T., Caporali, R. F., Verschueren, P., et al. (2026) · Annals of the Rheumatic Diseases · doi:10.1016/j.ard.2026.05.006 · Cited by 5
Sandner, E., Hu, B., Simiceanu, A., Fontana, L., Jakovljevic, I., Henriques, A., Wagner, A., & Gütl, C. (2024) · International Conference on Foundation and Large Language Models (FLLM) · doi:10.1109/fllm63129.2024.10852425 · Cited by 6
“According to estimates in the literature, reviewers spend an average of 0.9 minutes in the TiAb phase and 7 minutes in the FT phase to screen a single record [37].”
InfluentialExploring the definitions of gender bias in healthcare literature: a scoping review protocol
Bargeri, S., Schaap, L. S., Castellini, G., Gianola, S., Innocenti, T., Ostelo, R., Tomaiuolo, R., Vidal-Itriago, A., & Rubinstein, S. M. (2025) · MethodsX · doi:10.1016/j.mex.2025.103545 · Cited by 3
“Moreover, it has been reported that, although human reviewers are considered the “ gold standard, ” they are not perfect (average error rate due to false inclusion/exclusion of 10.76 %, varying by topic) [35].”
InfluentialMilella, F., Oltolini, M., & Bandini, S. (2024) · International Conference on PErvasive Technologies Related to Assistive Environments (PETRA) · doi:10.1145/3652037.3652046 · Cited by 4
COVID-19: Current Challenges and Future Perspectives
Leggat, P., Frean, J., & Blumberg, L. (2022) · MDPI eBooks · doi:10.3390/books978-3-0365-3119-9 · Cited by 4
Abògúnr̀in, Ṣ., Queiros, L., Bednarski, M., Sumner, M., Baehrens, D., & Witzmann, A. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.30.462652 · Cited by 3
Lai, H., Ma, N., Zhao, W., Liu, J., Pan, B., Tian, J., Chen, Y., Ma, B., Shang, H., Liu, J., et al. (2025) · Chinese Science Bulletin (Chinese Version) · doi:10.1360/csb-2025-5395 · Cited by 2
Artificial intelligence in medication management and drug dosing in critical care
Azarvash, N., Patel, D. A., Quickfall, D., Hambrick, H. R., KASHANI, K. B., & Tootooni, M. S. (2025) · Journal of Translational Critical Care Medicine · doi:10.1097/jtccm-d-25-00024 · Cited by 5
Zhang, X., Chang, Z., Zhao, S., Wu, X., Wang, X., Ai, G., & Ning, Z. (2026) · Frontiers in Nutrition · doi:10.3389/fnut.2026.1731627 · Cited by 4
“ASReview predicts study relevance by training a classification model on labeled abstracts and continuously reprioritizes the remaining records according to their likelihood of inclusion (23, 25, 26).”
Stuhlfauth, V., Saheb, Y., & Bouzouina, L. (2026) · Environmental Research Communications · doi:10.1088/2515-7620/ae4563 · Cited by 2
Screenathon 2.0: human–AI collaborative screening applied to patient-generated health data
Bergmann, J. G., Azzi, T. D. S., Neeleman, R. C., Monschau, K., Yazan, B., Jalsovec, E., Westerbeek, E., Weijdema, F., Bruin, J. D., Fang, Q., et al. (2026) · Scientific Reports · doi:10.1038/s41598-026-45385-5 · Cited by 1
Prasetia, A. C. G., & Urciuoli, L. (2026) · Business Strategy and the Environment · doi:10.1002/bse.70846 · Cited by 3
Hilkenmeier, F., Stoltenberg, M., & Stierle, C. M. G. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03228-4 · Cited by 8
“This process is not only cognitively demanding [69] even describe it as “over-whelming”), but also prone to inconsistencies and variability across human reviewers [8, 63, 81].”
Rowley, M. A., Barfield, W. R., Rivas, G. A., Reid, K. R., & Hartsock, L. A. (2024) · Current Orthopaedic Practice · doi:10.1097/bco.0000000000001266 · Cited by 2
“12 With machine learning, once topics have been established by the program, a researcher hoping to perform a systematic review can narrow their search for articles by choosing a speci fi c topic provided by the program and more accurately include or exclude papers.”
BackgroundPalermo, M. B., Policarpo, L. M., da Costa, C. A., & Righi, R. D. R. (2022) · Network Modeling Analysis in Health Informatics and Bioinformatics · doi:10.1007/s13721-022-00384-0 · Cited by 5
“We combined two approaches to ensure accuracy during the selection process: assessment of records by more than one reviewer (Gartlehner et al. 2020; Waffenschmidt et al. 2019; Wang et al. 2020), and crowdsourcing (Noel-Storr 2019; Nama et al.”
MethodTóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2023) · Research Square · doi:10.21203/rs.3.rs-3425312/v1 · Cited by 2
Boer, R. D., Hofstee, L., Leenaars, C., Bagheri, A., Teijema, J. J., & van de Schoot, R. (2024) · doi:10.31234/osf.io/zht9d · Cited by 1
Robleto, E., & Shehadeh, L. A. (2025) · American Journal of Physiology. Heart and Circulatory Physiology · doi:10.1152/ajpheart.00374.2025 · Cited by 2
Advani, S. D., Smith, A., Kalu, I. C., Perez, R., Hendren, S., Dantes, R., Edwards, J. R., Soe, M. M., Yi, S. H., Young, J., et al. (2024) · Antimicrobial Stewardship & Healthcare Epidemiology · doi:10.1017/ash.2024.397 · Cited by 1
Glenn, D. L., Choi, S. H., & Zimmerman, R. S. (2025) · Journal of Psychopharmacology · doi:10.1177/02698811251353251 · Cited by 1
“Prior research has demonstrated up to 95% reductions in screening workload while maintaining high recall rates of 70%–100% after screening only 10%–15% of records (Ferdinands et al., 2023; Gates et al., 2019; Miwa et al., 2014; Yu et al., 2018; Wang et al., 2020; Van De Schoot et al., 2021).”
Emotional wellbeing and language proficiencies in adolescence: A systematic review
Ticheloven, A., McMonagle, S., & Blom, E. (2025) · International Journal of Bilingualism · doi:10.1177/13670069251360172 · Cited by 3
“The ASReview algorithm iteratively identified potentially relevant abstracts, and abstracts were screened until 500 consecutive abstracts were rated as irrelevant, a method commonly used in review research (Wang et al., 2020).”
Myles, L. A. M., Hotton, M., Madden, F., & Salkovskis, P. M. (2025) · Clinical Psychology Review · doi:10.1016/j.cpr.2025.102661 · Cited by 1
Lockwood, K., & Wood, T. (2025) · Women s Health · doi:10.1177/17455057251367843 · Cited by 4
What we know about child maltreatment perpetrators: A scoping review of research focus and gaps
(Jip), J. J. V. G., van Koppen, M. V., Schoonmade, L. J., (Machteld), M. J. V., & Harte, J. M. (2025) · Aggression and Violent Behavior · doi:10.1016/j.avb.2025.102122 · Cited by 1
ASReview Dory: Bringing new and exciting models to ASReview LAB
van der Kuil, T., Teijema, J. J., Bruin, J. D., & van de Schoot, R. (2026) · Software Impacts · doi:10.1016/j.simpa.2025.100809 · Cited by 2
“The screening phase of systematic reviews is one of the most time-consuming tasks in the evidence synthesis process [1].”
Hough, B., Drever, N., & Manger, S. (2026) · American Journal of Lifestyle Medicine · doi:10.1177/15598276261419770 · Cited by 1
“55 Having two reviewers decreases this error rate.”
Versteeg, J., Bruin, M. L. D., Schermer, M., Najafabadi, S. N., Mitra, M., Leopold, C., Mantel-Teeuwisse, A. K., Goettsch, W., & Bloem, L. T. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag051 · Cited by 2
Fütterer, T., König, L., Campos, D., Scherer, R., Zitzmann, S., & Hecht, M. (2026) · Advances in Methods and Practices in Psychological Science · doi:10.1177/25152459261442150 · Cited by 2
“By comparison, traditional screening methods have been shown to result in error rates of approximately 10%, often because of fatigue and the extended duration of the process (Wang et al., 2020).”
Bloudek, L. M., Cichewicz, A. B., Patel, K., Sullivan, S. D., & Kallmes, K. M. (2026) · PharmacoEconomics · doi:10.1007/s40273-026-01648-7
Sabani, A., Farah, M. H., Catyanadika, P. E., Dewi, D. R. S., & Tawani, V. (2026) · Computers and Education Artificial Intelligence · doi:10.1016/j.caeai.2026.100667 · Cited by 8
“Spirit child” and concomitant paedicides in Africa: a systematic review of the empirical literature
Owusu, E. S. (2024) · International Journal of Comparative and Applied Criminal Justice · doi:10.1080/01924036.2024.2388238 · Cited by 3
“Thus, even though “[s]ingle screening is an efficient use of time and resources, . . . there is a higher risk of missing relevant studies” compared to double screening (Page, Moher, et al., 2021, p. 9; see also Wang et al., 2020; Waffenschmidt et al., 2019).”
Oami, T., Okada, Y., & Nakada, T. (2023) · medRxiv · doi:10.1101/2023.12.29.23300652 · Cited by 1
Oami, T., Okada, Y., & Nakada, T. (2024) · medRxiv · doi:10.1101/2024.05.22.24307740 · Cited by 2
Chen, Z., Liu, T., Mo, Y., Fu, Q., Lei, S., Tong, T., & Tang, X. (2025) · medRxiv · doi:10.1101/2025.09.18.25335860 · Cited by 3
Taheny, K., Johnston, B. M., & Ryan, K. (2021) · BMJ Supportive & Palliative Care · doi:10.1136/bmjspcare-2021-003451
Costa, L., Schlösser, T. P. C., Kruyt, M. C., & Castelein, R. M. (2022) · Journal of Clinical Medicine · doi:10.3390/jcm11040918
“Unfortunately, this kind of error is inherent to the methodology of title and abstract screening, which was performed by two independent researchers [3].”
BackgroundOami, T., Okada, Y., Fukuda, T., Sakuraya, M., Nakada, T., & Shime, N. (2022) · medRxiv · doi:10.1101/2022.11.17.22282374
“However, the process needs an iterative screening of a vast number of publications, which is often a time-consuming and intensive workload for reviewers [2, 3].”
BackgroundFeature selection analysis of systematic review citation screening algorithm
Li, S., Du, Y.-C., Yun, Y.-L., & Liu, H.-G. (2022) · IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) · doi:10.1109/imcec55388.2022.10020072 · Cited by 1
“Furthermore, the error rate of manual systematic review has been reported by Wang Z et al [11].”
MethodSucaldito, M. S. F. P., & Yu, K. C. (2024) · doi:10.2196/preprints.57648 · Cited by 1
Comparison of large language models for citation screening: A protocol for a prospective studyPreprint
Oami, T., Okada, Y., & Nakada, T. (2024) · medRxiv · doi:10.1101/2024.06.26.24309513
Xia, Z., Ye, J., Hu, B., Qiang, Q., & Debnath, R. (2024) · Research Square · doi:10.21203/rs.3.rs-5063165/v1 · Cited by 1
Instruments for Value Elucidation in Older Adults in Clinical Practice—A Scoping Review
van den Bosch, G., Versteeg, K. S., Metselaar, S., Koot, L., Burchell, G. L., Geerse, O. P., & Stoffels, J. M. J. (2025) · Journal of the American Geriatrics Society · doi:10.1111/jgs.19356
Conversational Agents in the Legal Domain: A Systematic Review of the Literature
Magni, M., & Milella, F. (2025) · Lecture notes in networks and systems · doi:10.1007/978-3-031-84460-7_12 · Cited by 1
Gil, D. F., José, S., Ascenso, A., Babič, M. N., Segal, E., Meletiadis, J., Gangneux, J., Weiskerger, C. J., Solo‐Gabriele, H. M., Valério, E., et al. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5148819
Zary, N. (2025) · Preprints.org · doi:10.20944/preprints202506.0749.v1
A scoping review of definitions and conceptualizations of ecosyndemics in syndemic researchPreprint
Gnaoui, Y., Slagboom, M. N., Jong, J. C. K., & Beenackers, M. A. (2025) · medRxiv · doi:10.1101/2025.07.24.25331762 · Cited by 1
“Human reviewers usually fail to identify 10% of the relevant papers (Wang et al., 2020), ASReview aims to find 95% of the relevant publications (ASReview, 2022).”
Chen, Z., Liu, T., Mo, Y., Fu, Q., Lei, S., Tong, T., & Tang, X. (2025) · doi:10.2196/preprints.85051
Kasakewitch, J. P. G., Eguchi, M., Perim, V., Lech, G. E., Balthazar, C. J. P., Nogueira, R., Malcher, F., & Lima, D. J. M. (2025) · Journal of Laparoendoscopic & Advanced Surgical Techniques · doi:10.1177/10926429251393122
Han, H. W., Kim, J. H., Chio, J., Shim, S. R., Lee, J., & Lee, M. (2025) · doi:10.2196/preprints.87065
Pharmacist-Led Optimization of High-Alert Medications in Critical Care Settings
Alreshidi, H. M. N., Alreshidi, A. A. T., Alshilash, I. A., Alhamili, M. N. M., Albalawi, A. O. S., Alasmari, S. A. S., Almsaudi, B. S., Albalawi, B. S., Alruwaili, M. A. S., Alenezi, K. A. T., et al. (2024) · International Journal of Computational and Experimental Science and Engineering · doi:10.22399/ijcesen.4484
“…instance, steering teams away from long-acting benzodiazepines like midazolam or lorazepam, which are strongly associated with delirium, towards more titratable agents like propofol or the α2-agonist dexmedetomidine, which has a demonstrated lower incidence of delirium in certain populations [20].”
Teo, L., Elswyk, M. E. V., Lau, C. S., & Shanahan, C. J. (2023) · Figshare · doi:10.6084/m9.figshare.24558168
Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26585199
Teo, L., Elswyk, M. E. V., Lau, C. S., & Shanahan, C. J. (2024) · Figshare · doi:10.6084/m9.figshare.24558213
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26725952
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26725949
Boetje, J., & van de Schoot, R. (2024) · Figshare · doi:10.6084/m9.figshare.26687522
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26725940
Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26585205
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26725943
Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26585202
Bourke, M., Haddara, A., Loh, A., Saravanamuttoo, K., Bruijns, B. A., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26725946
Bourke, M., Haddara, A., Loh, A., Carson, V., Breau, B., & Tucker, P. (2024) · Figshare · doi:10.6084/m9.figshare.26585208
Diekel, F., Arendt, R., & Berger, M. (2026) · MethodsX · doi:10.1016/j.mex.2026.103799
“For our review, we set the recall target at 90 %, based on reported error rates of 10 – 13 % in manual reviews [12,13], with a significance level of p 0.05, achieving p = 0.046 for both biodiversity and carbon respectively.”
InfluentialSystematic reviews in minutes to hours using artificial intelligencePreprint
Bakker, L., Caganek, T., Rooprai, A., & Hume, S. (2026) · medRxiv · doi:10.64898/2026.02.06.26345764
Affengruber, L., Emprechtinger, R., Persad, E., Kleijnen, J., & Gartlehner, G. (2026) · BMC Medical Research Methodology · doi:10.1186/s12874-026-02838-9
“[4] of relevant studies.”
Liebrand, B., Braunhofer, L., van der Weegen, W., Vissers, K. C. P., Goethem, J. W. V., Zundert, J. H. M. V., van den Boogaart, M., & van der Wal, S. (2026) · European Spine Journal · doi:10.1007/s00586-026-09925-x
Generative AI and adaptive systems for customising help-seeking scaffolds: A systematic review
Jecha, J., Igweani, C. C., Banaras, A., Liang, Y., & Mwalyego, F. M. (2026) · Information Services & Use · doi:10.1177/18758789261486037
Ahn, S., Kim, S., & Valdes, J. (2026) · Springer international handbooks of education · doi:10.1007/978-3-031-88898-4_24-1
Bernal-Córdoba, C., Valldecabres, A., Borona-Valencia, M., Canales-Muñoz, O., & Silva-del-Río, N. (2026) · Preventive Veterinary Medicine · doi:10.1016/j.prevetmed.2026.106908
Fernandez-Agudo, A., Lester, H., Tarazona, J. V., & Rivero‐Pino, F. (2026) · Frontiers in Toxicology · doi:10.3389/ftox.2026.1829326
“…at the abstract screening step, implying a more time-consuming process, spending more human resources on the elaboration of systematic review, which might be the basis for subsequent research proposals or recommendations and policies (Wang et al., 2020; Dempster et al., 2022; Focosi et al., 2021).”
Aygün, Y., Tüfekçi, Ş., Böke, H., Yagin, B., Berktaş, H., Canpolat, B., Turhan, F. H., Norman, G., & Güldoğan, E. (2026) · Frontiers in Psychology · doi:10.3389/fpsyg.2026.1762732
“Two reviewers independently screened all records retrieved from the database searches (Waffenschmidt et al., 2019; Wang et al., 2020).”
Xu, H., Liu, J., & Cheng, Q. (2026) · Applied Sciences · doi:10.3390/app16125827
Methods for assessing problem‐solving competencies in preschoolers: A scoping review
Elsen, J. V., Mæyer, S. D., & Torbeyns, J. (2026) · Review of Education · doi:10.1002/rev3.70181
Metadata
- Authors
- Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H.
- Affiliations
- Mayo Clinic (US)
- Corresponding author
- Wang, Zhen
- Venue
- PLOS ONE
- Publisher
- Public Library of Science (PLoS)
- Published
- 2020-01-14
- Volume
- 15
- Issue
- 1
- Pages
- e0227742
- ISSN
- 1932-6203 (ISSN-L)
- PMC
- PMC6959565
- OpenAlex
- W2999536597
- Semantic Scholar
- dd9e64171e0c3e867739daaf5c5023aed6e2e3ab
- Cited by
- 217
- References
- 24
- Citations by year
- 2026: 49 · 2025: 59 · 2024: 54 · 2023: 22 · 2022: 15 · 2021: 11 · 2020: 7
- FWCI
- 2.83
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- PLoS ONE (publishedVersion, cc-by)
- Europe PMC (PubMed Central) (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- systematic reviews, abstract screening, false exclusion, gold standard, full-text screening
- MeSH terms
- Abstracting and Indexing, Systematic Reviews as Topic, Algorithms, Electronic Data Processing, Humans, Research Design, Databases, Bibliographic
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
