Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews
Open accessPublished version of a preprint
Abstract
Background: We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr’s predictions varied by review or study-level characteristics. Methods: For a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining records, as predicted by the tool. We retrospectively simulated the liberal-accelerated screening approach. We estimated the time savings and proportion missed compared with dual independent screening. For reviews with pairwise meta-analyses, we evaluated changes to the pooled effects after removing the missed studies. We explored whether the tool’s predictions varied by review and study-level characteristics. Results: Using the ML-assisted liberal-accelerated approach, we wrongly excluded 0 to 3 (0 to 14%) records that were included in the final reports, but saved a median (IQR) 26 (9, 42) h of screening time. One missed study was included in eight pairwise meta-analyses in one systematic review. The pooled effect for just one of those meta-analyses changed considerably (from MD (95% CI) − 1.53 (− 2.92, − 0.15) to − 1.17 (− 2.70, 0.36)). Of 802 records in the final reports, 87% were correctly predicted as relevant. The correctness of the predictions did not differ by review (systematic or rapid, P = 0.37) or intervention type (simple or complex, P = 0.47). The predictions were more often correct in reviews with multiple (89%) vs. single (83%) research questions ( P = 0.01), or that included only trials (95%) vs. multiple designs (86%) ( P = 0.003). At the study level, trials (91%), mixed methods (100%), and qualitative (93%) studies were more often correctly predicted as relevant compared with observational studies (79%) or reviews (83%) ( P = 0.0006). Studies at high or unclear (88%) vs. low risk of bias (80%) ( P = 0.039), and those published more recently (mean (SD) 2008 (7) vs. 2006 (10), P = 0.02) were more often correctly predicted as relevant. Conclusion: Our screening approach saved time and may be suitable in conditions where the limited risk of missing relevant records is acceptable. Several of our findings are paradoxical and require further study to fully understand the tasks to which ML-assisted screening is best suited. The findings should be interpreted in light of the fact that the protocol was prepared for the funder, but not published a priori. Because we used a convenience sample, the findings may be prone to selection bias. The results may not be generalizable to other samples of reviews, ML tools, or screening approaches. The small number of missed studies across reviews with pairwise meta-analyses hindered strong conclusions about the effect of missed studies on the results and conclusions of systematic reviews.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Advances in automation and crowdsourcing have the potential to reduce screening time in rapid reviews.
“Nonetheless, forthcoming advances in automation (eg, active machine learning) 49 50 and crowdsourcing 8 51 have the potential to reduce screening time when conducting rapid reviews.”
From Updated recommendations for the Cochrane rapid review methods guidance for rapid reviews of effectiveness (2024) · cites this paper as [50]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · see citing works
PRISMA-P Group, Moher, D., Shamseer, L., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., Shekelle, P., & Stewart, L. A. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-1 · Cited by 26,223 · see citing works
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · PLoS Medicine · doi:10.1371/journal.pmed.0040296 · Cited by 23,020 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., Vandenbroucke, J. P., & Initiative, F. T. S. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-8-200710160-00010 · Cited by 17,796 · see citing works
Shea, B. J., Reeves, B. C., Wells, G., Thuku, M., Hamel, C., Moran, J., Moher, D., Tugwell, P., Welch, V., Kristjansson, E., et al. (2017) · BMJ · doi:10.1136/bmj.j4008 · Cited by 10,636 · see citing works
von Elm, E., Altman, D. G., Egger, M., Pocock, S. J., Gøtzsche, P. C., & Vandenbroucke, J. P. (2014) · International Journal of Surgery · doi:10.1016/j.ijsu.2014.07.013 · Cited by 9,909 · see citing works
An Empirical Evaluation of the System Usability Scale
Bangor, A., Kortum, P. T., & Miller, J. T. (2008) · International Journal of Human–Computer Interaction · doi:10.1080/10447310802205776 · Cited by 5,181 · see citing works
Evidence summaries: the evolution of a rapid review approach
Khangura, S., Konnyu, K., Cushman, R., Grimshaw, J., & Moher, D. (2012) · Systematic Reviews · doi:10.1186/2046-4053-1-10 · Cited by 1,232 · see citing works
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · see citing works
A scoping review of rapid review methods
Tricco, A. C., Antony, J., Zarin, W., Strifler, L., Ghassemi, M., Ivory, J., Perrier, L., Hutton, B., Moher, D., & Straus, S. E. (2015) · BMC Medicine · doi:10.1186/s12916-015-0465-6 · Cited by 1,146 · see citing works
International Journal of Social Research Methodology
(2000) · BRILL eBooks · doi:10.1163/9789004457751_016 · Cited by 1,080 · see citing works
How Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · see citing works
Page, M. J., Shamseer, L., Altman, D. G., Tetzlaff, J., Sampson, M., Tricco, A. C., Catalá-López, F., Li, L., Reid, E. K., Sarkis-Onofre, R., et al. (2016) · PLOS Medicine · doi:10.1371/journal.pmed.1002028 · Cited by 768 · see citing works
Borah, R., Brown, A. W., Capers, P. L., & Kaiser, K. A. (2017) · BMJ Open · doi:10.1136/bmjopen-2016-012545 · Cited by 737 · see citing works
A full systematic review was completed in 2 weeks using automation tools: a case study
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.008 · Cited by 687 · see citing works
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · see citing works
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638 · see citing works
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · see citing works
Systematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · see citing works
Marshall, I. J., Noel‐Storr, A., Kuiper, J., Thomas, J., & Wallace, B. C. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1287 · Cited by 435 · see citing works
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · see citing works
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 351 · see citing works
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · see citing works
On behalf of the founding members of the ICASR group, Beller, E., Clark, J., Tsafnat, G., Adams, C., Diehl, H., Lund, H., Ouzzani, M., Thayer, K., Thomas, J., et al. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0740-7 · Cited by 191 · see citing works
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184 · see citing works
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167 · see citing works
Gates, A., Johnson, C., & Hartling, L. (2018) · Systematic Reviews · doi:10.1186/s13643-018-0707-8 · Cited by 148 · see citing works
Self-citationPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148 · see citing works
Gates, A., Guitard, S., Pillay, J., Elliott, S. A., Dyson, M. P., Newton, A. S., & Hartling, L. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1222-2 · Cited by 133 · see citing works
Self-citationTsou, A. Y., Treadwell, J. R., Erinoff, E., & Schoelles, K. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01324-7 · Cited by 117 · cited 1× · see citing works
“To what extent ML-assisted methods could compromise the validity of systematic reviews’ findings is vitally important, but few studies have reported on this outcome [17, 18].”
MethodHow to conduct systematic reviews more expeditiously?
Tsertsvadze, A., Chen, Y.-F., Moher, D., Sutcliffe, P., & McCarthy, N. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0147-7 · Cited by 104 · see citing works
Can abstract screening workload be reduced using text mining? User experiences of the tool Rayyan
Olofsson, H., Brolund, A., Hellberg, C., Silverstein, R., Stenström, K., Österberg, M., & Dagerhamn, J. (2017) · Research Synthesis Methods · doi:10.1002/jrsm.1237 · Cited by 102 · see citing works
Gates, A., Gates, M., Sebastianski, M., Guitard, S., Elliott, S. A., & Hartling, L. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01031-w · Cited by 80 · see citing works
Self-citationAssessing the accuracy of machine-assisted abstract screening with DistillerAI: a user study
Gartlehner, G., Wagner, G., Lux, L. J., Affengruber, L., Dobrescu, A. I., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1221-3 · Cited by 73 · see citing works
O’Connor, A. M., Tsafnat, G., Gilbert, S. B., Thayer, K. A., & Wolfe, M. S. (2018) · Systematic Reviews · doi:10.1186/s13643-017-0667-4 · Cited by 69 · see citing works
Orellana-Paucar, A., & Vintimilla-Rojas, D. (2020) · Systematic Reviews · doi:10.1186/s13643-019-1259-2 · Cited by 53 · see citing works
Thomas, J. (2013) · OA Evidence-Based Medicine · doi:10.13172/2053-2636-1-2-1109 · Cited by 44 · see citing works
Improving the conduct of systematic reviews: a process mining perspective
Pham, B., Ebrahim Bagheri, A., Rios, P. D. L., Pourmasoumi, A., Robson, R. C., Hwee, J., Isaranuwatchai, W., Darvesh, N., Page, M. J., & Tricco, A. C. (2018) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2018.06.011 · Cited by 40 · see citing works
Shekelle, P., Shetty, K. D., Newberry, S. J., Maglione, M., & Motala, A. (2017) · Annals of Internal Medicine · doi:10.7326/l17-0124 · Cited by 38 · see citing works
Reddy, S. M., Patel, S. V., Weyrich, M. S., Fenton, J. J., & Viswanathan, M. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01450-2 · Cited by 38 · see citing works
Tsou, C., Robinson, S., Boyd, J., Jamieson, A., Blakeman, R., Bosich, K., Yeung, J., Waters, S., & Hendrie, D. V. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01349-y · Cited by 29 · see citing works
Gartlehner, G., Wagner, G., Lux, L. J., Affengruber, L., Dobrescu, A. I., Kaminski-Hartenthaler, A., & Viswanathan, M. (2019) · doi:10.23970/ahrqepcmethmachinedistiller · Cited by 3 · see citing works
Flagged first, then by citations
Research works printed but not linked: 1
- [35]Lefebvre C Glanville J Briscoe S Littlewood A Marshall C Metzendorf M-I JPT H Thomas J Chandler J Cumpston M Li T Page MJ Welch VA Chapter 4: Searching for and selecting study Cochrane Handbook for Systematic Reviews of Interventions, version 6.0 (updated July 2019). Cochrane 2019
Cited by
Sorted by citations
Garritty, C., Hamel, C., Trivella, M., Gartlehner, G., Nussbaumer-Streit, B., Devane, D., Kamel, C., Griebler, U., & King, V. J. (2024) · BMJ · doi:10.1136/bmj-2023-076335 · Cited by 260 · see citing works
Using artificial intelligence methods for systematic review in health sciences: A systematic review
Blaizot, A., Veettil, S. K., Saidoung, P., Moreno‐Garcia, C. F., Wiratunga, N., Aceves‐Martins, M., Lai, N. M., & Chaiyakunapruk, N. (2022) · Research Synthesis Methods · doi:10.1002/jrsm.1553 · Cited by 223 · see citing works
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 65 · see citing works
Affengruber, L., van der Maten, M. M., Spiero, I., Nußbaumer-Streit, B., Mahmić-Kaknjo, M., Ellen, M., Gooßen, K., Kantorová, L., Hooft, L., Riva, N., et al. (2024) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02320-4 · Cited by 57 · see citing works
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 53 · see citing works
Haby, M. M., Barreto, J. O. M., Kim, J. Y. H., Peiris, S., Mansilla, C., Torres, M., Guerrero‐Magaña, D. E., & Revéiz, L. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1664 · Cited by 47 · see citing works
Nama, N., Hennawy, M., Barrowman, N., O’Hearn, K., Sampson, M., & McNally, J. D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01632-6 · Cited by 50 · see citing works
Burgard, T., & Bittermann, A. (2023) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000509 · Cited by 29 · see citing works
Carey, N., Harte, M., & McCullagh, L. (2022) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2022.05.017 · Cited by 26 · see citing works
Digital Tools to Support the Systematic Review Process: An Introduction
Schmidt, L., Cree, I. A., Campbell, F., & WCT EVI MAP group (2025) · Journal of Evaluation in Clinical Practice · doi:10.1111/jep.70100 · Cited by 22 · see citing works
Chatbots for conducting systematic reviews in pediatric dentistry
Rokhshad, R., Mohammad, F. D., Nomani, M., Mohammad‐Rahimi, H., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105733 · Cited by 20 · see citing works
Machine learning for accelerating screening in evidence reviews
Chappell, M., Edwards, M., Watkins, D., Marshall, C., & Graziadio, S. (2023) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.12021 · Cited by 17 · see citing works
Oami, T., Okada, Y., Sakuraya, M., Fukuda, T., Shime, N., & Nakada, T. (2023) · Journal of Epidemiology · doi:10.2188/jea.je20230227 · Cited by 17 · see citing works
Ajiji, P., Cottin, J., Picot, C., Uzunali, A., Ripoche, E., Cucherat, M., & Maison, P. (2022) · European Journal of Clinical Pharmacology · doi:10.1007/s00228-022-03329-8 · Cited by 14 · see citing works
Do it faster with PICOS: Generative AI-Assisted systematic review screening
Vallamchetla, S. K., Abdelkader, O., Elnaggar, A., Ramadan, D., Islam Shourav, M. M., Riaz, I. B., & Lin, M. P. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104860 · Cited by 13 · see citing works
Torres, J. P. F., Mulligan, C. E. A., Jorge, J. A., & Moreira, C. (2025) · Information · doi:10.3390/info16050420 · Cited by 12 · see citing works
Large language models for the screening step in systematic reviews in dentistry
Rokhshad, R., Bagherianlemraski, M., Ehsani, S., Haghighat, S., & Schwendicke, F. (2025) · Journal of Dentistry · doi:10.1016/j.jdent.2025.105877 · Cited by 8 · see citing works
Semi-automating abstract screening with a natural language model pretrained on biomedical literature
Ng, S. H.-X., Teow, K. L., Ang, G. Y., Tan, W. S., & Hum, A. Y. M. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02353-8 · Cited by 7 · see citing works
Yao, X., Saha, A., Saravanan, S., Low, A., & SUSSMAN, J. S. (2025) · Cancer Innovation · doi:10.1002/cai2.70021 · Cited by 7 · see citing works
Automated synthesis of biodiversity knowledge requires better tools and standardised research output
Cornford, R., Millard, J. W., González‐Suárez, M., Freeman, R., & Johnson, T. F. (2022) · Ecography · doi:10.1111/ecog.06068 · Cited by 6 · see citing works
Artificial intelligence in systematic literature reviews: a case for cautious optimism
Kharawala, S., Mahajan, A., & Gandhi, P. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2021.03.012 · Cited by 6 · see citing works
Gashi, F., Kovačić, T., Gashi, A. I., Boshnjaku, A., & Shalaj, I. (2023) · Sports · doi:10.3390/sports11090187 · Cited by 5 · see citing works
Towards Automated Screening of Literature on Artificial Intelligence in Nursing
Moen, H., Alhuwail, D., Björne, J., Block, L., Celin, S., Jeon, E., Kreiner, K., Mitchell, J. L., Ozegovic, G., Ronquillo, C., et al. (2022) · Studies in health technology and informatics · doi:10.3233/shti220155 · Cited by 5 · see citing works
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2023) · Research Square · doi:10.21203/rs.3.rs-3425312/v1 · Cited by 2 · see citing works
Menéndez, M. M., Pico-Saltos, R., & Zambrano, C. E. (2024) · Future Business Journal · doi:10.1186/s43093-024-00339-z · Cited by 1 · see citing works
Comparing artificial intelligence versus human screening in systematic reviews
Gorici, M., Espin, A. T., & Oremus, M. (2026) · Health and Quality of Life Outcomes · doi:10.1186/s12955-026-02642-5 · Cited by 1 · see citing works
Mahlangu, G., Phahlane, M. M., & Mbohwa, C. (2025) · doi:10.2196/preprints.86347
Cartography of risks related to drug exposure during pregnancy
Ajiji, P. (2022) · theses.fr (ABES) · doi:10.70675/59762305ze937z47b1z94a5z1a103a1c89af
Zhang, Y., Zhao, Y., Zhang, P., Zhu, S., Tian, T., Zhang, Y., Li, D., Chen, T., Li, J., & Xie, Y. (2026) · Trials · doi:10.1186/s13063-026-09923-w
Barragán, A. M., Bonett, S. E. O., Rodríguez-Grande, E.-I., Orjuela-Cañón, Á. D., Perdómo, O., & Sánchez, G. (2026) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70098
Adam, G. P., Davies, M. C., Caputo, E. L., Mai, H. J., Coppola, E. L., Holmer, H. K., Kuhn, E., Balk, E. M., Trikalinos, T. A., Ivlev, I., et al. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10118
- Authors
- Gates, A., Gates, M., DaRosa, D., Elliott, S. A., Pillay, J., Rahman, S., Vandermeer, B., & Hartling, L.
- Affiliations
- University of Alberta (CA)
- Corresponding author
- Gates, Allison
- Venue
- Systematic Reviews
- Published
- 2020-11-27
- Volume
- 9
- Issue
- 1
- Pages
- 272-272
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC7694314
- OpenAlex
- W3104156511
- Cited by
- 31
- References
- 43
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 4 · 2023: 8 · 2022: 6 · 2021: 2
- FWCI
- 2.44
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- Agency for Healthcare Research and Quality (290-2015-00001-I)
- Canadian Institutes of Health Research
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › scientometrics and bibliometrics research
- Keywords
- machine learning, title and abstract screening, semi-automated screening, rapid review, pooled effect estimates, risk of bias, research design, research questions, prediction accuracy, funding sources, conflicts of interest
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Mass Screening, Research Design, Retrospective Studies
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
