Systematic review automation technologies
Open accessThe DOI matches its Crossref record; 136 of the 137 references it cites were found in a registry or an index.
Abstract
Systematic reviews, a cornerstone of evidence-based medicine, are not produced quickly enough to support clinical practice. The cost of production, availability of the requisite expertise and timeliness are often quoted as major contributors for the delay. This detailed survey of the state of the art of information systems designed to support or automate individual tasks in the systematic review, and in particular systematic reviews of randomized controlled clinical trials, reveals trends that see the convergence of several parallel research projects.We surveyed literature describing informatics systems that support or automate the processes of systematic review or each of the tasks of the systematic review. Several projects focus on automating, simplifying and/or streamlining specific tasks of the systematic review. Some tasks are already fully automated while others are still largely manual. In this review, we describe each task and the effect that its automation would have on the entire systematic review process, summarize the existing information system support for each task, and highlight where further research is needed for realizing automation for the task. Integration of the systems that automate systematic review tasks may lead to a revised systematic review workflow. We envisage the optimized workflow will lead to system in which each systematic review is described as a computer program that automatically retrieves relevant trials, appraises them, extracts and synthesizes data, evaluates the risk of bias, performs meta-analysis calculations, and produces a report in real time.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Figure 1 Existing methods for systematic reviews follow these steps with some variations. Not all systematic reviews follow all steps. This process typically takes between 12 and 24 months. Adapted from the Cochrane [10] and CREBP [11] Manuals for systematic reviews. SR systematic review, SD standard deviation.](https://figures.cochat.ai/paper-figures/0307e81b72d14502b2371df395b8abca/systematic-review-automation-technologies-figure-1.jpg)

Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“many of the tools have been developed independently as stand-alone software and are often not compatible with other tools”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [5]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“An effective parallelisation of SR processes needs to be supported by the use of a purposefully adapted computer technology [11, 12].”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [12]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“An increasing number of projects are underway which focus on the automation of segments of the systematic review process, and although several tools and software have been developed, so far, none of them span the entire process of review production”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [5]
The cited source says what the claim attributes to it.
In the citing paper
“Title and abstract screening is a review step that may be particularly amenable to automation or semi-automation [2–4].”
From Performance and usability of machine learning for screening in systematic reviews: a comparative evaluation of three tools (2019) · cites this paper as [3]
The cited source says what the claim attributes to it.
In the citing paper
“The authors illustrated that not all SR tasks are equally amenable to automation.”
From How to conduct systematic reviews more expeditiously? (2015) · cites this paper as [12]
4 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & for the PRISMA Group (2009) · BMJ · doi:10.1136/bmj.b2535 · Cited by 83,823
Bias in meta-analysis detected by a simple, graphical test
Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997) · BMJ · doi:10.1136/bmj.315.7109.629 · Cited by 56,722 · cited 1×
“, funnel plots [130] show whether publication bias is likely to affect the conclusion of the”
BackgroundPreferred reporting items for systematic reviews and meta-analyses: the PRISMA Statement.
(2009) · PubMed Central · Cited by 44,859
Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & the PRISMA Group* (2009) · Annals of Internal Medicine · doi:10.7326/0003-4819-151-4-200908180-00135 · Cited by 37,554
Scoping studies: towards a methodological framework
Arksey, H., & O'Malley, L. (2005) · International Journal of Social Research Methodology · doi:10.1080/1364557032000119616 · Cited by 36,742 · cited 1×
“Current systems Global evidence maps [18,19] and scoping studies [20] are both literature review methods designed to complement systematic review and identify research gaps.”
BackgroundCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement
Moher, D., Liberati, A., Tetzlaff, J., & Altman, D. G. (2009) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2009.06.005 · Cited by 19,575
Statistical Methods for Meta-Analysis
Wasserman, S., Hedges, L. V., & Olkin, I. (1988) · Journal of Educational Statistics · doi:10.2307/1164953 · Cited by 10,326
Cochrane Handbook for Systematic Reviews of Interventions
Tarsilla, M. (2010) · Journal of MultiDisciplinary Evaluation · doi:10.56645/jmde.v6i14.284 · Cited by 5,178
Users' guides to the medical literature
Guyatt, G. H. (1993) · JAMA · doi:10.1001/jama.270.17.2096 · Cited by 2,970
Greenhalgh, T., & Peacock, R. (2005) · BMJ · doi:10.1136/bmj.38636.593461.68 · Cited by 2,045 · cited 3×
“step, it has been shown to improve retrieval [115], and is a recommended practice [46].”
BackgroundQuantifying the burden of disease: the technical basis for disability-adjusted life years.
Murray, C. J. L. (1994) · Europe PMC (PubMed Central) · Cited by 1,916
Meta-DiSc: a software for meta-analysis of test accuracy data
Zamora, J., Abraira, V., Muriel, A., Khan, K., & Coomarasamy, A. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-31 · Cited by 1,883
Summing up: The Science of Reviewing Research
Wilkinson, I., Light, R. J., & Pillemer, D. B. (1987) · Journal of Educational Statistics · doi:10.2307/1164691 · Cited by 1,471
Duplicate Record Detection: A Survey
Elmagarmid, A. K., Ipeirotis, P. G., & Verykios, V. S. (2007) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2007.250581 · Cited by 1,267
Clinical Trial Registration: A Statement from the International Committee of Medical Journal Editors
De Angelis, C., Drazen, J. M., Frizelle, F. A., Haug, C., Hoey, J., Horton, R., Kotzin, S., Laine, C., Marusic, A., Overbeke, A. J. P. M., et al. (2004) · New England Journal of Medicine · doi:10.1056/nejme048225 · Cited by 1,241
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 2×
“This process is prolonged and subject to review, editing, and publication delays [4,7].”
BackgroundSelf-citationA Survey of Automatic Query Expansion in Information Retrieval
Carpineto, C., & Romano, G. (2012) · ACM Computing Surveys · doi:10.1145/2071389.2071390 · Cited by 1,109 · cited 1×
“Automatic Query Expansion (AQE) is the collective name for algorithms that modify the user's query before it is processed by the search engine [64].”
BackgroundHow Quickly Do Systematic Reviews Go Out of Date? A Survival Analysis
Shojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Doucette, S., & Moher, D. (2007) · Annals of Internal Medicine · doi:10.7326/0003-4819-147-4-200708210-00179 · Cited by 838 · cited 2×
“This process is prolonged and subject to review, editing, and publication delays [4,7].”
BackgroundRobinson, K. A., & Dickersin, K. (2002) · International Journal of Epidemiology · doi:10.1093/ije/31.1.150 · Cited by 833 · cited 1×
“, MeSH terms [49]) and search strategies [30,34,50] most suitable for the research question.”
BackgroundWhetzel, P. L., Noy, N. F., Shah, N. H., Alexander, P. R., Nyulas, C., Tudorache, T., & Musen, M. A. (2011) · Nucleic Acids Research · doi:10.1093/nar/gkr469 · Cited by 783 · cited 1×
“BioPortal [73] is an ontology that maps vocabularies and includes MeSH but still lacks mappings to EMTREE or other literary database vocabularies.”
BackgroundForest plots: trying to see the wood and the trees
Lewis, S., & Clarke, M. (2001) · BMJ · doi:10.1136/bmj.322.7300.1479 · Cited by 763 · cited 1×
“The most common way to summarize a meta-analysis is using a forest plot [129] but other plots exist, such as Galbraith diagrams (also called radial dia-”
MethodMorris, C. R. (2005) · JAMA · doi:10.1001/jama.294.1.81 · Cited by 763
Meta-Analyst: software for meta-analysis of binary, continuous and diagnostic data
Wallace, B. C., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2009) · BMC Medical Research Methodology · doi:10.1186/1471-2288-9-80 · Cited by 659
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.270.17.2093 · Cited by 655
Avoidable Waste in the Production and Reporting of Research Evidence
Chalmers, I., & Glasziou, P. (2009) · Obstetrics & Gynecology · doi:10.1097/aog.0b013e3181c3020d · Cited by 600 · cited 1×
“Task description The resources and time required to produce a systematic review are so great that independently creating a duplicate of an existing review is both wasteful and avoidable [26].”
BackgroundSelf-citationLiving Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596 · cited 1×
“A ‘living’ systematic review is updated at the time it is required to be used in practice [139].”
BackgroundHaynes, R. B., & Wilczynski, N. L. (2004) · BMJ · doi:10.1136/bmj.38068.557998.ee · Cited by 547
Deploying an interactive machine learning system in an evidence-based practice center: abstrackr
Wallace, B. C., Small, K., Brodley, C. E., Lau, J., & Trikalinos, T. A. (2012) · Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium · doi:10.1145/2110363.2110464 · Cited by 532 · cited 3×
“Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
BackgroundMethodAn international registry of systematic-review protocols
Booth, A., Clarke, M., Ghersi, D., Moher, D., Petticrew, M., & Stewart, L. (2011) · The Lancet · doi:10.1016/s0140-6736(10)60903-8 · Cited by 512 · cited 2×
“Accommodation in systematic review registries such as PROSPERO and standard data exchanges such as HL7 [140] may assist in the standardization and quality assurance of automatically generated systematic reviews.”
BackgroundFormulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · cited 2×
“Task description There is no single correct way to formulate review questions [16]; although, it is desirable to prioritize review questions by burden of disease for which there is a lack of review in the area.”
BackgroundUsers' Guides to the Medical Literature: I. How to Get Started
Oxman, A. D. (1993) · JAMA · doi:10.1001/jama.1993.03510170083036 · Cited by 469
Lowe, H. J. (1994) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.1994.03510380059038 · Cited by 468 · cited 2×
“Some have specialized query languages that may include logical operators such as OR, AND, and NOT, syntax for querying specific fields such as ‘authors’ and ‘year’, operators such as ADJ (adjacent to) and NEAR, and controlled keyword vocabularies such as Medical Subject Heading (MeSH) [49] for PubMed, and EMTREE for E…”
BackgroundChecking reference lists to find additional studies for systematic reviews
Horsley, T., Dingwall, O., & Sampson, M. (2011) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.mr000026.pub2 · Cited by 443 · cited 2×
“A review on checking reference lists to find additional studies for systematic reviews also found that citation tracking identify additional studies [115].”
BackgroundOptimal search strategies for retrieving systematic reviews from Medline: analytical survey
Montori, V. M., Wilczynski, N. L., Morgan, D., & Haynes, R. B. (2005) · BMJ · doi:10.1136/bmj.38336.804167.47 · Cited by 422 · cited 3×
“Searching for clinical trials can be supported by specifically crafted filters and strategies [29-34].”
BackgroundMethodSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 3×
“Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
BackgroundMethodParsCit: an Open-source CRF Reference String Parsing Package
Councill, I. G., Giles, C. L., & Kan, M. (2008) · Language Resources and Evaluation · Cited by 326
Evidence & Gap Maps: A tool for promoting evidence informed policy and strategic research agendas
Snilstveit, B., Vojtkova, M., Bhavsar, A., Stevenson, J., & Gaarder, M. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2016.05.015 · Cited by 323
The Global Evidence Mapping Initiative: Scoping research in broad topic areas
Bragge, P., Clavisi, O., Turner, T., Tavender, E., Collie, A., & Gruen, R. L. (2011) · BMC Medical Research Methodology · doi:10.1186/1471-2288-11-92 · Cited by 310 · cited 1×
“Current systems Global evidence maps [18,19] and scoping studies [20] are both literature review methods designed to complement systematic review and identify research gaps.”
BackgroundSumming Up: The Science of Reviewing Research.
Hays, W. L. (1985) · PsycCRITIQUES · doi:10.1037/023434 · Cited by 307 · cited 1×
“Task description For a systematic review to be useful to a clinician, it needs to be presented in a clear, precise and informative manner [127,128].”
BackgroundWilczynski, N. L., Haynes, R. B., & the Hedges Team (2004) · BMC Medicine · doi:10.1186/1741-7015-2-23 · Cited by 291
Web question answering: is more always better?
Dumais, S., Banko, M., Brill, E., Lin, J., & Ng, A. (2002) · Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/564376.564428 · Cited by 289 · cited 1×
“For this task, a form of an automatic web-based question-answering system [38,39] in which”
MethodBoyack, K. W., Newman, D., Duhon, R. J., Klavans, R., Patek, M., Biberstine, J. R., Schijvenaars, B., Skupin, A., Ma, N., & Börner, K. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018029 · Cited by 271 · cited 1×
“Examples of document similarity measures include ones based on the terms that appear in the documents [66], and ones based”
Backgroundfor the ORIENT study investigators, Imai, E., Chan, J. C. N., Ito, S., Yamasaki, T., Kobayashi, F., Haneda, M., & Makino, H. (2011) · Diabetologia · doi:10.1007/s00125-011-2325-z · Cited by 236 · cited 1×
“’) [123] and the number of events in each arm in another (‘In the olmesartan group, 116 developed the primary outcome (41.”
BackgroundScaling question answering to the web
Kwok, C., Etzioni, O., & Weld, D. S. (2001) · ACM Transactions on Information Systems · doi:10.1145/502115.502117 · Cited by 230
The Handbook of Computational Linguistics and Natural Language Processing
(2010) · Wiley eBooks · doi:10.1002/9781444324044 · Cited by 228
Wong, S. S.-L., Wilczynski, N. L., & Haynes, R. B. (2004) · PubMed · Cited by 227
Proceedings of the 30th Annual ACM Symposium on Applied Computing
(2015) · doi:10.1145/2695664 · Cited by 214
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214 · cited 2×
“Often, information highlighting algorithms focus on PICO elements reported in the trial [95-97] but may also focus on the size of the trial [97,98], and its randomization procedure [95,97].”
BackgroundMethodApplications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212 · cited 2×
“Current systems Decision support systems use NLP to automatically highlight sentences and phrases that reviewers are likely to use for appraisal [93,94].”
BackgroundA systematic comparison of software dedicated to meta-analysis of causal studies
Bax, L., Yu, L.-M., Ikeda, N., & Moons, K. G. (2007) · BMC Medical Research Methodology · doi:10.1186/1471-2288-7-40 · Cited by 210
Defrosting the Digital Library: Bibliographic Tools for the Next Generation Web
Hull, D., Pettifer, S. R., & Kell, D. B. (2008) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1000204 · Cited by 209
Are systematic reviews up-to-date at the time of publication?
Beller, E. M., Chen, J. K.-H., Wang, U. L.-H., & Glasziou, P. P. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-36 · Cited by 196 · cited 6×
“The most commonly used databases in systematic reviews are the general databases MEDLINE, Cochrane Library, and EMBASE [55] but hundreds of specialty databases also exists, e.”
BackgroundInfluentialSelf-citationExaCT: automatic extraction of clinical trial characteristics from journal publications
Kiritchenko, S., de Bruijn, B., Carini, S., Martin, J., & Sim, I. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-56 · Cited by 188 · cited 3×
“Often, information highlighting algorithms focus on PICO elements reported in the trial [95-97] but may also focus on the size of the trial [97,98], and its randomization procedure [95,97].”
BackgroundMethodTransactions on Engineering Technologies: World Congress on Engineering and Computer Science 2015
(2017) · CERN Document Server (European Organization for Nuclear Research) · doi:10.1007/978-981-10-2717-8 · Cited by 180
Taking advantage of the explosion of systematic reviews: an efficient MEDLINE search strategy.
Shojania, K. G., & Bero, L. A. (2001) · PubMed · Cited by 176
Answering English questions by computer: a survey
Simmons, R. F. (1965) · Communications of the ACM · doi:10.1145/363707.363732 · Cited by 173 · cited 1×
“Question answering systems have been in development since the 1960s [37].”
Background[The International Clinical Trials Registry Platform - ICTRP].
Stegemann, H. (2007) · PubMed · Cited by 161
Handbook of Computational Linguistics and Natural Language Processing
Clark, A., Fox, C., & Lappin, S. (2010) · Cited by 159
Supporting Systematic Reviews Using Text Mining
Ananiadou, S., Rea, B., Okazaki, N., Procter, R., & Thomas, J. (2009) · Social Science Computer Review · doi:10.1177/0894439309332293 · Cited by 144 · cited 2×
“Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
BackgroundMethodThe librarian's roles in the systematic review process: a case study.
Harris, M. R. (2005) · Europe PMC (PubMed Central) · Cited by 138
Interventions to increase adherence to medications for tobacco dependence
Hollands, G. J., McDermott, M. S., Lindson-Hawley, N., Vogt, F., Farley, A., & Aveyard, P. (2015) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.cd009164.pub2 · Cited by 135
The automation of systematic reviews
Tsafnat, G., Dunn, A., Glasziou, P., & Coiera, E. (2013) · BMJ · doi:10.1136/bmj.f139 · Cited by 131 · cited 4×
“However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [3-9].”
BackgroundInfluentialSelf-citationAll trials must be registered and the results published
Chalmers, I., Glasziou, P., & Godlee, F. (2013) · BMJ · doi:10.1136/bmj.f105 · Cited by 131 · cited 1×
“Studies that investigated the efficacy of trial registration found that only about half of the clinical trials are published [57] and not all published trials are registered [58].”
BackgroundSelf-citationAutomatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 2×
“Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
BackgroundMethodRetrieving Clinical Evidence: A Comparison of PubMed and Google Scholar for Quick Clinical Searches
Shariff, S. Z., Bejaimal, S. A., Sontrop, J. M., Iansavichus, A. V., Haynes, R. B., Weir, M. A., & Garg, A. X. (2013) · Journal of Medical Internet Research · doi:10.2196/jmir.2624 · Cited by 119
Evidence Gap Maps — A Tool for Promoting Evidence-Informed Policy and Prioritizing Future Research
Snilstveit, B., Vojtkova, M., Bhavsar, A., & Gaarder, M. (2013) · World Bank, Washington, DC eBooks · doi:10.1596/1813-9450-6725 · Cited by 115 · cited 1×
“Current systems Global evidence maps [18,19] and scoping studies [20] are both literature review methods designed to complement systematic review and identify research gaps.”
BackgroundProceedings of the 2nd ACM SIGHIT International Health Informatics Symposium
(2012) · doi:10.1145/2110363 · Cited by 112
Find Duplicates among the PubMed, EMBASE, and Cochrane Library Databases in Systematic Review
Qi, X., Yang, M., Ren, W., Jia, J., Wang, J., Han, G., & Fan, D. (2013) · PLoS ONE · doi:10.1371/journal.pone.0071838 · Cited by 98 · cited 2×
“, EndNote [82], ProCite [83]) already have automatic search for duplicate records to semi-automate citation deduplication quite accurately [78,79].”
BackgroundDocument clustering of scientific texts using citation contexts
Aljaber, B., Stokes, N., Bailey, J., & Pei, J. (2009) · Information Retrieval · doi:10.1007/s10791-009-9108-x · Cited by 96 · cited 1×
“Examples of document similarity measures include ones based on the terms that appear in the documents [66], and ones based on papers they cite [67].”
BackgroundDéjà vu—A study of duplicate citations in Medline
Errami, M., Hicks, J. M., Fisher, W., Trusty, D., Wren, J. D., Long, T. C., & Garner, H. R. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btm574 · Cited by 95 · cited 1×
“Many plagiarism detection algorithms have mechanisms to understand paper structure and find similarities between papers [90-92].”
BackgroundOptimal search strategies for detecting health services research studies in MEDLINE
Wilczynski, N. L. (2004) · Canadian Medical Association Journal · doi:10.1503/cmaj.1040512 · Cited by 94 · cited 2×
“Searching for clinical trials can be supported by specifically crafted filters and strategies [29-34].”
BackgroundSo many filters, so little time: the development of a search filter appraisal checklist
Glanville, J., Bayliss, S., Booth, A., Dundar, Y., Fernandes, H., Fleeman, N. D., Foster, L., Fraser, C., Fry-Smith, A., Golder, S., et al. (2008) · Journal of the Medical Library Association : JMLA · doi:10.3163/1536-5050.96.4.011 · Cited by 87 · cited 3×
“Searching for clinical trials can be supported by specifically crafted filters and strategies [29-34].”
BackgroundEMBASE versus MEDLINE for family medicine searches: can MEDLINE searches find the forest or a tree?
Wilkins, T., Gillies, R. A., & Davies, K. J. (2005) · PubMed Central · Cited by 86
Medline and Embase complement each other in literature searches
Woods, D., & Trewheellar, K. (1998) · BMJ · doi:10.1136/bmj.316.7138.1166 · Cited by 85
Huser, V., & Cimino, J. J. (2013) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2012-001501 · Cited by 85 · cited 1×
“Studies that investigated the efficacy of trial registration found that only about half of the clinical trials are published [57] and not all published trials are registered [58].”
BackgroundSystematic Reviews and Librarians
McKibbon, A. (2006) · Library Trends · doi:10.1353/lib.2006.0049 · Cited by 84 · cited 1×
“Therefore, reviewers require specific knowledge of dozens of literary and non-literary databases, each with its own search engine, metadata, and vocabulary [53-55].”
BackgroundSmith, G. T., & Wells, N. E. J. (1979) · Monographs · Cited by 82
Coiera, E. (1998) · BMJ · doi:10.1136/bmj.317.7171.1469 · Cited by 78 · cited 1×
“The full automation of systematic reviews that will deliver the best evidence at the right time to the point of care is the logical next step [14].”
BackgroundSelf-citationCrossCheck: an effective tool for detecting plagiarism
ZHANG, H. (. (2010) · Learned Publishing · doi:10.1087/20100103 · Cited by 70 · cited 1×
“Many plagiarism detection algorithms have mechanisms to understand paper structure and find similarities between papers [90-92].”
BackgroundBalk, E. M., Chung, M., Chen, M. L., Chang, L. K. W., & Trikalinos, T. A. (2013) · Systematic Reviews · doi:10.1186/2046-4053-2-97 · Cited by 70 · cited 1×
“Language bias remains an unanswered problem with little existing systems available to address it [45].”
BackgroundSim, I., Tu, S. W., Carini, S., Lehmann, H. P., Pollock, B. H., Peleg, M., & Wittkowski, K. M. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.11.002 · Cited by 68
Rethinking credible evidence synthesis
Doshi, P., Jones, M., & Jefferson, T. (2012) · BMJ · doi:10.1136/bmj.d7898 · Cited by 60 · cited 1×
“Task description The biomedical literature is the main source of evidence for systematic reviews; however, some research studies can only be found outside of literature databases - the so called grey literature - and sometimes not at all [51,52].”
BackgroundCohen, A. M., Ambert, K., & McDonagh, M. (2012) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-12-33 · Cited by 57 · cited 3×
“Automation approaches are to use ML to infer exclusion and inclusion rules by observing a human screener [102-106], a class of algorithms called supervised ML.”
BackgroundMethodA multicomponent decision tool for prioritising the updating of systematic reviews
Takwoingi, Y., Hopewell, S., Tovey, D., & Sutton, A. J. (2013) · BMJ · doi:10.1136/bmj.f7191 · Cited by 57 · cited 1×
“However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [3-9].”
BackgroundProceedings of the 1st ACM International Health Informatics Symposium
Veinot, T. C., Çatalyürek, Ü. V., Luo, G., Andrade, H. C. M., & Smalheiser, N. R. (2010) · Cited by 55
EpiphaNet: An Interactive Tool to Support Biomedical Discoveries
Cohen, T., Whitfield, G. K., Schvaneveldt, R. W., Mukund, K., & Rindflesch, T. (2010) · Journal of Biomedical Discovery and Collaboration · doi:10.5210/disco.v5i0.3090 · Cited by 54 · cited 1×
“in the field of artificial intelligence on hypothesis generation and finding [22-24] may provide an automatic means of”
BackgroundEysenbach, J. T. (2001) · Medical Informatics and the Internet in Medicine · doi:10.1080/14639230110075459 · Cited by 53 · cited 2×
“Task description The biomedical literature is the main source of evidence for systematic reviews; however, some research studies can only be found outside of literature databases - the so called grey literature - and sometimes not at all [51,52].”
Background‘HypothesisFinder:’ A Strategy for the Detection of Speculative Statements in Scientific Text
Malhotra, A., Younesi, E., Gurulingappa, H., & Hofmann-Apitius, M. (2013) · PLoS Computational Biology · doi:10.1371/journal.pcbi.1003117 · Cited by 51
Modernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B. C., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 51 · cited 2×
“Proposed decision support systems for systematic reviewers include ones that help the basic tasks of systematic reviews [6,12,13].”
BackgroundAutomatic Summarization of Results from Clinical Trials
Summerscales, R. L., Argamon, S., Bai, S., Hupert, J., & Schwartz, A. (2011) · 2011 IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm.2011.72 · Cited by 49 · cited 2×
“We have identified only one study that attempted automatic extraction and synthesis of discrete distributions to NTT and RR formats [118].”
BackgroundMethodSmith, R., & Chalmers, I. (2001) · BMJ · doi:10.1136/bmj.323.7327.1437 · Cited by 49 · cited 3×
“However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [3-9].”
BackgroundUsing the Turning Research Into Practice (TRIP) database: how do clinicians really search?*
Meats, E., Brassey, J., Heneghan, C. J., & Glasziou, P. (2007) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.95.2.156 · Cited by 47 · cited 4×
“The Turning Research Into Practice (TRIP) [72] data-base uses multiple query fields including to search PubMed.”
MethodInfluentialSelf-citationKawamoto, K., Honey, A., & Rubin, K. H. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3123 · Cited by 47 · cited 1×
“Accommodation in systematic review registries such as PROSPERO and standard data exchanges such as HL7 [140] may assist in the standardization and quality assurance of automatically generated systematic reviews.”
BackgroundSmalheiser, N. R., Lin, C., Jia, L., Jiang, Y., Cohen, A. M., Yu, C., Davis, J. M., Adams, C. E., McDonagh, M. S., & Meng, W. (2014) · Health Information Science and Systems · doi:10.1186/2047-2501-2-1 · Cited by 47 · cited 1×
“Federated search engines need to translate the reviewer's query into the source databases' specific query languages and combine the results [68-70].”
BackgroundBeyond Trial Registration: A Global Trial Bank for Clinical Trial Reporting
Sim, I., & Detmer, D. E. (2005) · PLoS Medicine · doi:10.1371/journal.pmed.0020365 · Cited by 46 · cited 2×
“Automatic extraction can populate such databases [43].”
BackgroundCohen, A. M., Adams, C. E., Davis, J. M., Yu, C., Yu, P. S., Meng, W., Duggan, L., McDonagh, M., & Smalheiser, N. R. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1883046 · Cited by 45 · cited 1×
“The process of creating a systematic review can be broken down into between four and 15 tasks depending on resolution [5,10,13,15].”
BackgroundA Computational Model of Reasoning from the Clinical Literature
Rennels, G. D. (1987) · Lecture notes in medical informatics · doi:10.1007/978-3-642-93363-9 · Cited by 44
Robinson, K. A., Dunn, A. G., Tsafnat, G., & Glasziou, P. (2014) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2013.11.015 · Cited by 41
Self-citationInformatics and hypothesis‐driven research
Smalheiser, N. R. (2002) · EMBO Reports · doi:10.1093/embo-reports/kvf164 · Cited by 41 · cited 1×
“in the field of artificial intelligence on hypothesis generation and finding [22-24] may provide an automatic means of”
BackgroundRule-based deduplication of article records from bibliographic databases
Jiang, Y., Lin, C., Meng, W., Yu, C., Cohen, A. M., & Smalheiser, N. R. (2014) · Database · doi:10.1093/database/bat086 · Cited by 41 · cited 1×
“Heuristic [84], machine learning (ML) algorithms and probabilistic string matching are used to compare author names, journal, conference or book names (known as venue) and titles of citations.”
MethodQahtani, A. M. A., & Hajri, A. (2011) · Therapeutics and Clinical Risk Management · doi:10.2147/tcrm.s16415 · Cited by 40 · cited 1×
“, ‘duration of the first stage of labor’) [124] and only a reference to it can be found in the results section (e.”
BackgroundArchitecture for Knowledge-Based and Federated Search of Online Clinical Evidence
Coiera, E., Walther, M., Nguyen, K., & Lovell, N. H. (2005) · Journal of Medical Internet Research · doi:10.2196/jmir.7.5.e52 · Cited by 39 · cited 6×
“Quick Clinical's AQE algorithm enriches keywords differently depending on which field they were typed in [70].”
BackgroundMethodInfluentialSelf-citationHansen, M. J., Rasmussen, N. Ø., & Chung, G. (2008) · Journal of Telemedicine and Telecare · doi:10.1258/jtt.2008.007007 · Cited by 38 · cited 1×
“Often, information highlighting algorithms focus on PICO elements reported in the trial [95-97] but may also focus on the size of the trial [97,98], and its randomization procedure [95,97].”
BackgroundKuper, H., Nicholson, A., & Hemingway, H. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-4 · Cited by 36
Cohen, K. B., Demner‐Fushman, D., Friedman, C., Hirschman, L., & Pestian, J. P. (2007) · Cited by 36
A knowledge-based approach to citation extraction
Day, M.-Y., Tsai, R. T., Sung, C.-L., Lee, C., Wu, S.-H., Ong, C., & Hsu, W.-L. (2005) · IEEE International Conference on Information Reuse and Integration (IRI) · doi:10.1109/iri-05.2005.1506448 · Cited by 33 · cited 2×
“Current systems Automatic citation extraction tools [85-89] are core to snowballing systems.”
BackgroundFigure mining for biomedical research
Rodriguez-Esteban, R., & Iossifov, I. (2009) · Bioinformatics · doi:10.1093/bioinformatics/btp318 · Cited by 32 · cited 1×
“Systems that do make use of such information have been proposed [94,113] but have not yet seen adoption in this domain.”
Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries
Webber, B., & Joshi, A. (2012) · The Association for Computational Linguistics · Cited by 31
Hsu, W. W. Y., Speier, W. F., & Taira, R. K. (2012) · PubMed · Cited by 30
Sajja, P. S., & Akerkar, R. (2012) · Chapman & Hall/CRC eBooks · doi:10.1201/b12118-8 · Cited by 28
Systematic reviews: Work that needs to be done and not to be done
Adams, C. E., Polzmacher, S., & Wolff, A. (2013) · Journal of Evidence-Based Medicine · doi:10.1111/jebm.12072 · Cited by 28 · cited 1×
“However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [3-9].”
BackgroundWilczynski, N. L., McKibbon, K. A., & Haynes, R. B. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2011.04.007 · Cited by 26 · cited 1×
“filter that works well for finding systematic reviews but is limited to PubMed and DARE currently [36].”
BackgroundA computational model of reasoning from the clinical literature
Rennels, G. D., Shortliffe, E. H., Stockdale, F. E., & Miller, P. L. (1987) · Computer Methods and Programs in Biomedicine · doi:10.1016/0169-2607(87)90025-3 · Cited by 25
Time to Update and Quantitative Changes in the Results of Cochrane Pregnancy and Childbirth Reviews
Jaidee, W., Moher, D., & Laopaiboon, M. (2010) · PLoS ONE · doi:10.1371/journal.pone.0011553 · Cited by 24 · cited 1×
“However, even the conduct and upkeep of updated systematic reviews required to answer a significant proportion of clinical questions, is beyond our means without automation [3-9].”
BackgroundMeasuring the Performance of the Cochrane Library
Bero, L., Busuttil, G., Farquhar, C., Koehlmoos, T. P., Moher, D., Nylenna, M., Smith, R., & Tovey, D. (2012) · Cochrane Database of Systematic Reviews · doi:10.1002/14651858.ed000048 · Cited by 24 · cited 1×
“Global evidence maps are conducted in a formal process, similar to systematic reviews, and thus may take in excess of 2 years [21].”
BackgroundLong-Term Patterns of Online Evidence Retrieval Use in General Practice: A 12-Month Study
Magrabi, F., Westbrook, J. I., Kidd, M. R., Day, R. O., & Coiera, E. W. (2008) · Journal of Medical Internet Research · doi:10.2196/jmir.974 · Cited by 23 · cited 1×
“This interface not only prompts users to elaborate on their questions, it also guides inexpert users on how to ask clinical questions that will provide them with meaningful results [71].”
BackgroundSelf-citationA study of structured clinical abstracts and the semantic classification of sentences
Chung, G. Y., & Coiera, E. (2007) · Proceedings of the Workshop on BioNLP 2007 Biological, Translational, and Clinical Language Processing - BioNLP '07 · doi:10.3115/1572392.1572415 · Cited by 22 · cited 1×
“Current systems Decision support systems use NLP to automatically highlight sentences and phrases that reviewers are likely to use for appraisal [93,94].”
BackgroundSelf-citationIdentifying treatments, groups and outcomes in medical abstracts
Summerscales, R. (2009) · Touro Scholar (Touro College) · Cited by 22
CitePlag: A Citation-based Plagiarism Detection System Prototype
Meuschke, N., Gipp, B., & Breitinger, C. (2012) · doi:10.5281/zenodo.3483087 · Cited by 20
A comparison of metadata extraction techniques for crowdsourced bibliographic metadata management
Granitzer, M., Hristakeva, M., Jack, K., & Knight, R. (2012) · ACM Symposium on Applied Computing · doi:10.1145/2245276.2245462 · Cited by 20 · cited 2×
“Current systems Automatic citation extraction tools [85-89] are core to snowballing systems.”
BackgroundExploiting classification correlations for the extraction of evidence-based practice information.
(2012) · PubMed Central · Cited by 16
Evidence-based Medicine Search: a customizable federated search engine
Bracke, P. J., Howse, D., & Keim, S. M. (2008) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.96.2.108 · Cited by 16 · cited 1×
“Federated search engines need to translate the reviewer's query into the source databases' specific query languages and combine the results [68-70].”
BackgroundComputational Reasoning across Multiple Models
Tsafnat, G., & Coiera, E. W. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3023 · Cited by 14
Self-citationDuplicates in systematic reviews: A critical, but often neglected issue
Qi, X.-S. (2013) · World Journal of Meta-Analysis · doi:10.13105/wjma.v1.i3.97 · Cited by 13 · cited 2×
“, EndNote [82], ProCite [83]) already have automatic search for duplicate records to semi-automate citation deduplication quite accurately [78,79].”
BackgroundAn Analytical Assessment on Document Clustering
-, P., & Chatterjee, R. (2012) · International Journal of Computer Network and Information Security · doi:10.5815/ijcnis.2012.05.08 · Cited by 11
The Information Retrieval Process
Ceri, S., Bozzon, A., Brambilla, M., Valle, E. D., Fraternali, P., & Quarteroni, S. (2013) · Springer eBooks · doi:10.1007/978-3-642-39314-3_2 · Cited by 8 · cited 1×
“Task description Limitations of keyword searching have led to the development of secondary analysis of search re-sults [114].”
BackgroundA rule-based approach for automatic identification of publication types of medical papers
Sarker, A., & Mollá, D. (2010) · Australasian Document Computing Symposium · Cited by 8
Improving Open Information Extraction for Informal Web Documents with Ripple-Down Rules
Kim, M. H., & Compton, P. (2012) · Lecture notes in computer science · doi:10.1007/978-3-642-32541-0_14 · Cited by 7 · cited 1×
“Heuristic and ML systems can be complementary, and combinations of these approaches have proven accurate in other domains [108].”
BackgroundAabenhus, R. M., Jensen, J. S., & Cals, J. (2014) · British Journal of General Practice · doi:10.3399/bjgp14x679615 · Cited by 7 · cited 1×
“they should only be included in meta-analysis as one trial [81].”
BackgroundAustralian New Zealand Clinical Trials Registry: history and growth
Askie, L. M. (2011) · Journal of Evidence-Based Medicine · doi:10.1111/j.1756-5391.2011.01147.x · Cited by 6
Cepeda, M. S., Lobanov, V. S., & Berlin, J. A. (2013) · Journal of Pain · doi:10.1016/j.jpain.2013.06.005 · Cited by 6 · cited 1×
“The search strategy describes what keywords will be used in searches, which databases will be searched [10], if and how citations will be tracked [46,47], and what and how evidence will be identified in non-database sources (such as reference checking and expert contacts) [48].”
BackgroundSysBank: a knowledge base for systematic reviews of randomized clinical trials.
Carini, S., & Sim, I. (2003) · PubMed · Cited by 5
Chung, G. Y., & Coiera, E. W. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-48 · Cited by 5
Self-citationWolf, F. M., Grum, C. M., Bara, A., Milan, S. J., & Jones, P. W. (1995) · Lancaster EPrints (Lancaster University) · Cited by 3
Linking Citations to their Bibliographic references
Nhat, H. H., & Bysani, P. (2012) · Cited by 2
Research works printed but not linked: 15
- 1. Sackett DL, Straus S, Richardson WS, Rosenberg W, Haynes RB. Evidence-based Medicine: How to Teach and Practice EBM. Edinburgh: Churchill Livingstone; 2000.
- 11. Glasziou P, Beller E, Thorning S, Vermeulen M. Manual for systematic reviews. Gold Coast, Queensland, Australia: Centre for Research on Evidence Based Medicine; 2013.
- 40. Collaboration RTC. Review Manager (RevMan). 4.2 for Windows. Oxford, England: The Cochrane Collaboration; 2003.
- 41. Higgins J, Churchill R, Tovey D, Lasserson T, Chandler J. Update on the MECIR project: methodological expectations for Cochrane intervention. Cochrane Meth. 2011;suppl. 1:2.
- 82. REUTERS T. EndNote®. Bibliographies Made Easy™; 2011. http://www.endnote.com.
- 83. ResearchSoft I. ProCite Version 5. Berkeley, CA: Institute for Scientific Information; 1999.
- 88. Saleem O, Latif S. Proceedings of the World Congress on Engineering and Computer Science. San Francisco, USA; 2012. Information extraction from research papers by data integration and data validation from multiple header extraction sources.
- 99. Thomas J, Brunton J, Graziosi S. EPPI-Reviewer 4.0: software for research synthesis. EPPI-Centre, Social Science Research Unit, Institute of Education, University of London UK; 2010.
- 100. DistillerSR. http://www.systematic-review.net.
- 110. PubMed Central. http://www.ncbi.nlm.nih.gov/pmc.
- 111. Google Scholar. http://scholar.google.com.
- 120. Huwaldt JA. Plot Digitizer. 2009. http://plotdigitizer.sf.net.
- 121. Egner J, Mitchell M, Richter T, Winchen T. Engauge Digitizer: convert graphs or map files into numbers. 2013. http://digitizer.sf.net/
- 122. Rohatgi A. WebPlotDigitizer. 2013. http://arohatgi.info/WebPlotDigitizer/
- 133. Rosenberg MS, Adams DC, Gurevitch J. MetaWin: statistical software for meta-analysis. Massachusetts, USA: Sinauer Associates Sunderland; 2000.
Cited by
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
“An increasing number of projects are underway which focus on the automation of segments of the systematic review process, and although several tools and software have been developed, so far, none of them span the entire process of review production [5].”
BackgroundCochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews
Garritty, C., Gartlehner, G., Nussbaumer-Streit, B., King, V. J., Hamel, C., Kamel, C., Affengruber, L., & Stevens, A. (2021) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.10.007 · Cited by 1,141
“However, the methodological rigor and process that makes SR evidence trustworthy often take one to 2 years to complete [1 e 4] and limit their utility to meet the time-sensitive needs of stakeholders.”
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“The use of automation to assist in study selection is possibly the most advanced of all the areas where automation in systematic reviews is being developed; but others range from writing sections of the report, formulating the review question and automated data extraction and quality assessment [90-93].”
BackgroundA full systematic review was completed in 2 weeks using automation tools: a case study
Clark, J., Glasziou, P., Del Mar, C., Bannach-Brown, A., Stehlik, P., & Scott, A. M. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.01.008 · Cited by 687
Self-citationSearching for and selecting studies
Lefebvre, C., Glanville, J., Briscoe, S., Littlewood, A., Marshall, C., Metzendorf, M., Noel‐Storr, A., Rader, T., Shokraneh, F., Thomas, J., et al. (2019) · Cochrane Handbook for Systematic Reviews of Interventions · doi:10.1002/9781119536604.ch4 · Cited by 647
Metadata
- Authors
- Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E.
- Affiliations
- UNSW Sydney (AU)
- Bond University (AU)
- Corresponding author
- Tsafnat, Guy
- Venue
- Systematic Reviews
- Published
- 2014-07-09
- Volume
- 3
- Issue
- 1
- Pages
- 74-74
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC4100748
- OpenAlex
- W2095601341
- Cited by
- 472
- References
- 137
- Citations by year
- 2026: 41 · 2025: 42 · 2024: 56 · 2023: 44 · 2022: 44 · 2021: 68 · 2020: 46 · 2019: 36 · 2018: 32 · 2017: 24 · 2016: 16 · 2015: 18 · 2014: 5
- FWCI
- 5.75
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- Bond University Research Portal (Bond University) (publishedVersion, cc-by)
- PubMed (publishedVersion)
- e-publications@bond (Bond University) (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Health Sciences › Medicine › Public Health, Environmental and Occupational Health › Clinical practice guidelines implementation
- Keywords
- systematic review automation, information systems, randomized controlled trials, evidence-based medicine, risk of bias assessment, data extraction, meta-analysis, system integration
- MeSH terms
- Electronic Data Processing, Review Literature as Topic, Information Storage and Retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.