The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
The cited source says what the claim attributes to it.
In the citing paper
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
The cited source says what the claim attributes to it.
In the citing paper
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 622
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- 2005, BMC bioinformatics [electronic resource], 6, S16
- 1980, PROGRAM, 14, 127
Cited by
Sorted by citations
Items 101–200 of 411
An SVM-based high-quality article classifier for systematic reviews
Kim, S., & Choi, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.10.005 · Cited by 39
Natukunda, A., & Muchene, L. K. (2023) · Systematic Reviews · doi:10.1186/s13643-022-02163-4 · Cited by 45
“So far, results from existing automated screening algorithms estimate an average success rate between 30 and 70% in reducing the number of documents that have to be screened manually, albeit accompanied with some loss of potentially relevant studies [17, 18, 20].”
Automatic classification of literature in systematic reviews on food safety using machine learning
van den Bulk, L. M., Bouzembrak, Y., Gavai, A., Liu, N., van den Heuvel, L. J., & Marvin, H. J. P. (2022) · Current Research in Food Science · doi:10.1016/j.crfs.2021.12.010 · Cited by 38
“Timsina et al. (2016) retested four data sets used by Cohen et al. (2006) using three types of SVMs (linear, polynomial and evolutionary), NB and a single-layer NN.”
BackgroundMethodTeijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Vizan, P., van den Brand, S., Bockting, C., van de Schoot, R., et al. (2023) · Frontiers in Research Metrics and Analytics · doi:10.3389/frma.2023.1178181 · Cited by 36
“The third study compares the classifier performance in terms of work saved over sampling (Cohen et al., 2006) for different combinations of classifiers with feature extraction techniques and compares these combinations with the newly developed 17-layer CNN model.”
MethodMachine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36
“Another example is the voting perceptron-based automated citation classification system discussed in Cohen et al. (2006). They employed varying learning weights to penalize for misclassification of the relevant class; A drawback is that there is no universal way for choosing the most optimal weights across all reviews…”
MethodLiu, J., Timsina, P., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-016-9724-0 · Cited by 35
“Cohen et al. (2006), in a National Institute of Health (NIH) supported project, developed-a perceptron-based classifiers to identify journal articles for inclusion in systematic review update, based on the title and abstract of the articles.”
BackgroundMethodInfluentialNama, N., Iliriani, K., Xia, M. Y., Chen, B. P., Zhou, L. L., Pojsupap, S., Kappel, C., O’Hearn, K., Sampson, M., Menon, K., et al. (2017) · Translational Pediatrics · doi:10.21037/tp.2016.12.01 · Cited by 35
“The ability of the crowd to retain 100% of eligible studies throughout the entire screening process in our study exceeds the 95% cut-off utilized in the computerized automated text screening literature to identify promising algorithms (12,26).”
MethodMoreno‐García, C. F., Jayne, C., Elyan, E., & Aceves‐Martins, M. (2023) · Decision Analytics Journal · doi:10.1016/j.dajour.2023.100162 · Cited by 37
Ji, X., Machiraju, R., Ritter, A., & Yen, P. (2015) · PubMed Central · Cited by 34
Self-citationAdvanced analytics for the automation of medical systematic reviews
Timsina, P., Liu, J., & El-Gayar, O. (2016) · Information Systems Frontiers · doi:10.1007/s10796-015-9589-7 · Cited by 33
“…the Unified Medical Language Systems (UMLS) to extract medical terms as features for article classification, while the majority of existing research uses the “ bag-of-words ” approach (Adeva et al. 2014; Shemilt et al. 2013; Bekhuis and Demner-Fushman 2012; Wallace et al. 2010; Cohen et al. 2006).”
BackgroundMethodResultInfluentialKönig, L., Zitzmann, S., Fütterer, T., Campos, D., Scherer, R., & Hecht, M. (2024) · Research Synthesis Methods · doi:10.1002/jrsm.1762 · Cited by 38
Seed-driven Document Ranking for Systematic Reviews in Evidence-Based Medicine
Lee, G. E., & Sun, A. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3209994 · Cited by 33
“An evaluation measure called Work Saved over Sampling (WSS) was proposed in [3] for estimating how many documents can be excluded from the screening process.”
BackgroundMethodMeasuring the impact of screening automation on meta-analyses of diagnostic test accuracy
Norman, C. R., Leeflang, M., Porcher, R., & Névéol, A. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1162-x · Cited by 34
Burgard, T., & Bittermann, A. (2023) · Zeitschrift für Psychologie · doi:10.1027/2151-2604/a000509 · Cited by 29
(2010) · Cited by 32
Dalal, S. R., Shekelle, P., Hempel, S., Newberry, S. J., Motala, A., & Shetty, K. D. (2012) · Medical Decision Making · doi:10.1177/0272989x12457243 · Cited by 31
“Our results are in accord with prior attempts using machine learning to facilitate comparative effectiveness review data collection.(12,13,18) For example, an active learning model achieved 50% workload reductions and 100% sensitivity in several cases.”
MethodResultCohen, A. M. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.008177 · Cited by 32
Self-citationVarghese, A., Cawley, M. A., & Hong, T. (2017) · Environment Systems & Decisions · doi:10.1007/s10669-017-9670-5 · Cited by 33
“Machine learning has shown promise as a technique for limiting the number of documents to be screened manually in the context of scientific literature search (Frunza et al. 2011; Cohen et al. 2006; Aphinyanaphongs et al. 2005; Shemilt et al. 2014; Bekhuis and Demner-Fushman 2012; Wallace et al. 2010) and to determine…”
BackgroundPerlman‐Arrow, S., Loo, N., Bobrovitz, N., Yan, T., & Arora, R. K. (2023) · Research Synthesis Methods · doi:10.1002/jrsm.1636 · Cited by 30
“[19, 20] Some studies have assessed efficiency measures, such as impact to reviewer workload or time saved while screening, but these are typically conducted retrospectively, with data from completed reviews.”
BackgroundBoosting efficiency in a clinical literature surveillance system with LightGBM
Lokker, C., Abdelkader, W., Bagheri, E., Parrish, R., Cotoi, C., Navarro, T., Germini, F., Linkins, L., Haynes, R. B., Chu, L., et al. (2024) · PLOS Digital Health · doi:10.1371/journal.pdig.0000299 · Cited by 37
Deng, Z., Yin, K., Bao, Y., Armengol, V. D., Wang, C., Tiwari, A., Barzilay, R., Parmigiani, G., Braun, D., & Hughes, K. S. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00043 · Cited by 31
A question-answering framework for automated abstract screening using large language models
Akinseloyin, O., Jiang, X., & Palade, V. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae166 · Cited by 41
“25 For each SR, we calculated the performance metrics.”
MethodSemi-Automated evidence synthesis in health psychology: current methods and future prospects
Marshall, I. J., Johnson, B. T., Wang, Z., Rajasekaran, S., & Wallace, B. (2020) · Health Psychology Review · doi:10.1080/17437199.2020.1716198 · Cited by 30
“For evaluating classi fi er performance in the context of citation screening (as described above), custom metrics have been de fi ned that attempt to better capture the speci fi c trade-o ff s involved (Cohen et al., 2006; Wallace, Small, Brodley, & Trikalinos, 2010).”
MethodKontonatsios, G., Spencer, S., Matthew, P. W., & Korkontzelos, I. (2020) · Expert Systems with Applications X · doi:10.1016/j.eswax.2020.100030 · Cited by 30
Goldilocks: Just-Right Tuning of BERT for Technology-Assisted Review
Yang, E., MacAvaney, S., Lewis, D. D., & Frieder, O. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_34 · Cited by 32
A Multi-Channel Convolutional Neural Network approach to automate the citation screening process
van Dinter, R., Catal, C., & Teki̇nerdoğan, B. (2021) · Applied Soft Computing · doi:10.1016/j.asoc.2021.107765 · Cited by 33
Automating risk of bias assessment for clinical trials
Marshall, I. J., Kuiper, J., & Wallace, B. (2014) · ACM International Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB) · doi:10.1145/2649387.2649406 · Cited by 28
“has previously been recognized that machine learning tools can assist abstract screening [15]–[18], data extraction [19]–[23], summarization [24], [25], and scoping [26].”
BackgroundAbogunrin, S., Muir, J. M., Zerbini, C., & Sarri, G. (2025) · Frontiers in Pharmacology · doi:10.3389/fphar.2025.1454245 · Cited by 27
Systematic global stocktake of over 50,000 urban climate change studies
Montfort, S., Callaghan, M., Creutzig, F., Lamb, W. F., Lu, C., Repke, T., Ge, K., & Minx, J. C. (2025) · Nature Cities · doi:10.1038/s44284-025-00260-8 · Cited by 36
“The latter are increasingly entering climate change research, for instance, through recent methodological advances that make systematic evidence synthesis constantly updat-able 22 , and the development and application of AI-assisted screening of the literature for systematic reviews 36 .”
Extended Boolean retrieval for systematic biomedical reviews
Pohl, S., Zobel, J., & Moffat, A. (2010) · Cited by 27
Keshava, C., Davis, J. A., Stanek, J., Thayer, K. A., Galizia, A., Keshava, N., Gift, J. S., Vulimiri, S. V., Woodall, G. M., Gigot, C., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105956 · Cited by 26
“Screening continued until SWIFT Active indicated that it was likely that at least 95% of the relevant studies were identified, a percent identification often used to evaluate the performance of machine learning applications and considered comparable to human error rates (Bannach-Brown, 2018; Howard et al., 2016; Cohen…”
BackgroundImproving the utility of MeSH® terms using the TopicalMeSH representation
Yu, Z., Bernstam, E. V., Cohen, T., Wallace, B., & Johnson, T. R. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.03.013 · Cited by 25
“For both tasks, we used 15 publicly available systematic drug review corpora [34].”
BackgroundMethodMartin, P. B., Surian, D., Bashir, R., Bourgeois, F. T., & Dunn, A. G. (2019) · JAMIA Open · doi:10.1093/jamiaopen/ooy062 · Cited by 24
Orel, E., Ciglenečki, I., Thiabaud, A., Temerev, A., Calmy, A., Keiser, O., & Merzouki, A. (2023) · Journal of Medical Internet Research · doi:10.2196/39736 · Cited by 32
“…identified; PPV and NPV), recall (number of relevant papers identified using LiteRev among those identified using manual review), and WSS [33,34] were computed and discussed. where true negatives are the number of nonrelevant abstracts that were correctly identified as nonrelevant by…”
A Text-Mining Framework for Supporting Systematic Reviews.
Li, D., Wang, Z., Wang, L., Sohn, S., Shen, F., Murad, M. H., & Liu, H. (2016) · PubMed · Cited by 23
Choosing the Right Tool for the Job: Screening Tools for Systematic Reviews in Education
Zhang, Q., & Neitzel, A. (2024) · Journal of Research on Educational Effectiveness · doi:10.1080/19345747.2023.2209079 · Cited by 28
“For example, some new screening tools were equipped with machine learning-based text classification algorithms to rank the relevancy of the studies based on human reviewers ’ screening patterns, which significantly reduce resources required for conducting reviews (Cohen et al., 2006).”
Zero-shot Generative Large Language Models for Systematic Review Screening Automation
Wang, S., Scells, H., Zhuang, S., Potthast, M., Koopman, B., & Zuccon, G. (2024) · Lecture Notes in Computer Science · doi:10.1007/978-3-031-56027-9_25 · Cited by 26
“Lastly, we gauge the efficiency of automatic document screening using the Work Saved by Sampling at a specific recall level ( WSS ) [16].”
MethodAutomatic endpoint detection to support the systematic review process.
Blake, C., & Lucic, A. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.05.004 · Cited by 22
“Much of the research since then has focused on informatics tools that support the selection (information retrieval) stage of the process [3–7] with some work on the extraction stage [2,9].”
MethodCallaghan, M., Müller-Hansen, F., Bond, M., Hamel, C., Devane, D., Kusa, W., O’Mara-Eves, A., Spijker, R., Stevenson, M., Stansfield, C., et al. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02699-7 · Cited by 26
Learning Disentangled Representations of Texts with Application to Biomedical Abstracts
Jain, S., Banner, E., van de Meent, J.-W., Marshall, I. J., & Wallace, B. (2018) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/d18-1497 · Cited by 22
“In particular, we assess whether the induced representations afford improved retrieval of abstracts relevant to a particular systematic review (Cohen et al., 2006; Wal-lace et al., 2010).”
BackgroundMethodInfluentialThe use of bibliography enriched features for automatic citation screening
Olorisade, B. K., Brereton, P., & Andras, P. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103202 · Cited by 21
“(d) WSS: given a certain recall level (rl), the WSS is the percentage of the articles initially returned by the literature search which the researcher would not have to read because they have been screened out by the model [21].”
BackgroundMethodResultInfluentialThe challenge of high recall in biomedical systematic search
Karimi, S., Zobel, J., Pohl, S., & Scholer, F. (2009) · Data and Text Mining in Bioinformatics · doi:10.1145/1651318.1651338 · Cited by 21
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · Intelligent Systems with Applications · doi:10.1016/j.iswa.2023.200193 · Cited by 21
“The following models were used: A: Cohen et al. (2006), B: Matwin et al. (2010), C: Cohen (2008), D: Howard et al. (2016), E: Kontonatsios et al. (2020), F: van Dinter, Catal et al. (2021), G: Kusa et al. (2022 an FPR level of 0.35, obtains a lower AUC score.”
InfluentialASReview LAB v.2: Open-source text screening with multiple agents and a crowd of experts
Bruin, J. D., Lombaers, P., Kaandorp, C., Teijema, J. J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · Patterns · doi:10.1016/j.patter.2025.101318 · Cited by 27
“At that point, metrics such as recall, loss, time to discovery, 27 or work saved over sampling (WSS@95) 53 can be calculated.”
Neural Rankers for Effective Screening Prioritisation in Medical Systematic Review Literature Search
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2022) · Australasian Document Computing Symposium · doi:10.1145/3572960.3572980 · Cited by 22
“Intuitively, WSS@k measures the fraction of the screening workload saved if one stops examining the ranking once k% of the relevant documents have been found compared to screening the entire set of documents [5].”
BackgroundLokker, C., Bagheri, E., Abdelkader, W., Parrish, R., Afzal, M., Navarro, T., Cotoi, C., Germini, F., Linkins, L. A., Haynes, R. B., et al. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104384 · Cited by 22
Facilitating biomedical systematic reviews using ranked text retrieval and classification
Martínez, D., Karimi, S., Cavedon, L., & Baldwin, T. (2008) · Cited by 20
Cawley, M. A., Beardslee, R., Beverly, B., Hotchkiss, A. K., Kirrane, E. F., Sams, R. L., Varghese, A., Wignall, J., & Cowden, J. (2019) · Environment International · doi:10.1016/j.envint.2019.105228 · Cited by 20
Muthu, S. (2023) · International Orthopaedics · doi:10.1007/s00264-022-05672-y · Cited by 20
“Almost 86(±3)% of the relevant records were included with screening 30% of the total records for screening (RRF-30).”
BackgroundInfluentialAutomatically finding relevant citations for clinical guideline development
Bui, D. D. A., Jonnalagadda, S., & Fiol, G. D. (2015) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2015.09.003 · Cited by 19
“investigated a machine learning approach to solve a classification problem in drug effectiveness reviews [12,17].”
BackgroundBannach‐Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/255760 · Cited by 19
Li, D., Wu, L., Zhang, M., Shpyleva, S., Lin, Y.-C., Huang, H., Li, T., & Xu, J. (2024) · Frontiers in Drug Safety and Regulation · doi:10.3389/fdsfr.2024.1379260 · Cited by 19
A decision support system for automating document retrieval and citation screening
van Dinter, R., Catal, C., & Teki̇nerdoğan, B. (2021) · Expert Systems with Applications · doi:10.1016/j.eswa.2021.115261 · Cited by 21
Yang, J. J., Cohen, A. M., Cohen, A. M., & McDonagh, M. S. (2008) · PubMed Central · Cited by 18
Self-citationUsing MEDLINE Elemental Similarity to Assist in the Article Screening Process for Systematic Reviews
Ji, X., & Yen, P. (2015) · JMIR Medical Informatics · doi:10.2196/medinform.3982 · Cited by 18
“These measures are commonly used for evaluating similar work [7,12,15].”
InfluentialSelf-citationKim, S., & Choi, J. (2012) · Healthcare Informatics Research · doi:10.4258/hir.2012.18.1.18 · Cited by 19
“This process is described in greater detail in earlier studies [5,9].”
BackgroundResultSurian, D., Dunn, A. G., Orenstein, L., Bashir, R., Coiera, E. W., & Bourgeois, F. T. (2018) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2018.01.008 · Cited by 17
“While the results of studies testing semi-automated screening of published articles for inclusion in systematic reviews are not directly comparable, standard tools in this area use active learning to avoid screening 80% of articles to reach 95% recall [20,25,26].”
BackgroundMethodSimulation-based Active Learning for Systematic Reviews: A Scoping Review of LiteraturePreprint
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2023) · doi:10.31234/osf.io/67zmt · Cited by 17
“Noteworthy collections evaluated in this study include the Cohen dataset [71] and the CLEF datasets [39, 47], which are frequently employed in simulation research.”
Peron, M., Arena, S., Paltrinieri, N., Sgarbossa, F., & Boustras, G. (2022) · Risk Analysis · doi:10.1111/risa.14010 · Cited by 17
“For instance, literature research revealed automated classification focused on medical (Cohen et al., 2006; Timsina et al., 2016) and biomedical (Wallace et al., 2010) sectors, cloud manufacturing (Ellwein et al., 2020; Lolli et al., 2022; Sgarbossa et al., 2021), food safety (van den Bulk et al.,…”
BackgroundAdaptive search query generation and refinement in systematic literature review
Badami, M., Benatallah, B., & Báez, M. (2023) · Information Systems · doi:10.1016/j.is.2023.102231 · Cited by 19
Potential application of artificial intelligence in cancer therapy
Riaz, I. B., Khan, M. A., & Haddad, T. C. (2024) · Current Opinion in Oncology · doi:10.1097/cco.0000000000001068 · Cited by 19
Classifying Biomedical Abstracts Using Committees of Classifiers and Collective Ranking Techniques
Kouznetsov, A., Matwin, S., Inkpen, D. Z., Razavi, A. H., Frunza, O., Sehatkar, M., Seaward, L., & O’Blenis, P. (2009) · Lecture notes in computer science · doi:10.1007/978-3-642-01818-3_29 · Cited by 17
“This experiment demonstrated that Complement Naïve Bayes outperformed the Voting Perceptron results reported by [9] on 12 of the 15 datasets Regarding our experimental results on the data provided by TrialStat (that data set that we use in this paper), the obtained performance level, in terms of the achieved savings,…”
BackgroundMethodResultInfluentialMachine Learning assisted systematic reviewing in orthopaedics
Pijls, B. G. (2024) · Journal of Orthopaedics · doi:10.1016/j.jor.2023.11.051 · Cited by 16
A Reproducibility and Generalizability Study of Large Language Models for Query Generation
Staudinger, M., Kusa, W., Piroi, F., Lipani, A., & Hanbury, A. (2024) · Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (SIGIR-AP) · doi:10.1145/3673791.3698432 · Cited by 22
Chi-square-based Scoring Function for Categorization of MEDLINE Citations
Kastrin, A., Peterlin, B., & Hristovski, D. (2010) · Methods of Information in Medicine · doi:10.3414/me09-01-0009 · Cited by 16
“DC machinery has also been used in maintaining systematic reviews in the domain of internal medicine [40] and of the efficacy of drug therapy [41].”
MethodKusa, W., Hanbury, A., & Knoth, P. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_39 · Cited by 16
Efficient Extended Boolean Retrieval
Pohl, S., Moffat, A., & Zobel, J. (2011) · IEEE Transactions on Knowledge and Data Engineering · doi:10.1109/tkde.2011.63 · Cited by 17
ASReview: Open Source Software for Efficient and Transparent Active Learning for Systematic Reviews.Preprint
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2020) · arXiv (Cornell University) · Cited by 15
Nykvist, B., Macura, B., Xylia, M., & Olsson, E. (2025) · Environmental Evidence · doi:10.1186/s13750-025-00360-x · Cited by 17
“Finally, we also calculated Specificity (S), Precision (P), Recall (R) and Work Saved over Sampling (WSS) (according to Cohen et al. [21]).”
InfluentialTeijema, J. J., Hofstee, L., Brouwer, M., de Bruin, J., Ferdinands, G., de Boer, J., Siso, P. V., van den Brand, S. A. G. E., Bockting, C., van de Schoot, R., et al. (2022) · doi:10.31234/osf.io/t7bpd · Cited by 14
Artificial Intelligence Software to Accelerate Screening for Living Systematic Reviews
Fuller‐Tyszkiewicz, M., Jones, A., Vasa, R., Vasa, R., Grbin, L., Macdonald, J. A., Deane, C., Samuel, D., Evans-Whipp, T., & Olsson, C. A. (2025) · Clinical Child and Family Psychology Review · doi:10.1007/s10567-025-00519-5 · Cited by 19
“This evaluation approach is often referred to as work saved over sampling (WSS; Cohen et al., 2006), and the target proportion of relevant articles to retrieve is often set at 95% of total relevant articles within the full set of articles from one’s database search.”
Lee, T.-F., Hsieh, Y.-W., Yang, P., Tseng, C.-H., Lee, S., Yang, J., Chang, L., Wu, J., Tseng, C.-D., & Chao, P. (2024) · Radiation Oncology · doi:10.1186/s13014-023-02381-7 · Cited by 15
Evaluating GPT Models for Automated Literature Screening in Wastewater-Based Epidemiology
Chibwe, K., Mantilla‐Calderon, D., & Ling, F. (2024) · ACS Environmental Au · doi:10.1021/acsenvironau.4c00042 · Cited by 15
“29 Thus, while literature screening tasks can be formulated either way, the classifier approach lends the advantage of not generating nonsensical answers, thus making a more streamlined process.”
Research literature clustering using diffusion maps
Nieminen, P., Pölönen, I., & Sipola, T. (2013) · Journal of Informetrics · doi:10.1016/j.joi.2013.08.004 · Cited by 17
“Keywords: knowledge discovery process, literature mapping, data mining, clustering, diffusion map”
BackgroundTimsina, P., Liu, J., El-Gayar, O., & Shang, Y. (2016) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2016.151 · Cited by 15
“[9] used 50% training and 50% validation data, Adeva et al.”
MethodDocFlow: A Visual Analytics System for Question-Based Document Retrieval and Categorization
Qiu, R., Tu, Y., Wang, Y.-S., Yen, P., & Shen, H. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3219762 · Cited by 16
Self-citationLi, J., Kabouji, J., Bouhadoun, S., Tanveer, S., Filion, K. B., Gore, G. C., Josephson, C. B., Kwon, C., Jetté, N., Bauer, P. R., et al. (2023) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2023.07.010 · Cited by 15
Neeleman, R. C., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02472-w · Cited by 12
“[10], the number of Extra Relevant Records found at 10% recall, and the Average Time to Discover (ATD) a relevant record [14].”
BackgroundMethodAutomatic Literature Mapping Selection: Classification of Papers on Industry Productivity
Bispo, G. D., Vergara, G. F., Saiki, G. M., Martins, P. H. D. S., Coelho, J. G., Rodrigues, G. A. P., de Oliveira, M. N., Mosquéra, L. R., Gonçalves, V. P., Neumann, C., et al. (2024) · Applied Sciences · doi:10.3390/app14093679 · Cited by 15
Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization
Zhang, Y., Lease, M., & Wallace, B. (2017) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p17-2024 · Cited by 16
Towards minimizing the annotation cost of certified text classification
Bagdouri, M., Webber, W., Lewis, D. D., & Oard, D. W. (2013) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2505515.2505708 · Cited by 14
“…statistically valid estimates of classifier effectiveness is not, however, unique to e-discovery, but can be found in other areas where classifier results are subject to controversy, such as text classification in evidence-based medicine [4] and microarray classification in cancer diagnosis [24].”
BackgroundClassification of Research Citations (CRC)Preprint
Butt, B. H., Rafi, M., Jamal, A., Rehman, R. S. U., Alam, S. M. Z., & Alam, M. B. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1506.08966 · Cited by 14
“With the help of CRC, research scholar can easily classify the type of reading materials, along with other researcher's point of view in citing the same paper.”
BackgroundWard, C., Phiri, E., Goodman, C. A., Mipando, A. L. N., Malata, M. P., Manda, W. C., Mwapasa, V., & Powell‐Jackson, T. (2024) · Health Policy and Planning · doi:10.1093/heapol/czae112 · Cited by 25
Bag of words and embedding text representation methods for medical article classification
Cichosz, P. (2023) · International Journal of Applied Mathematics and Computer Science · doi:10.34768/amcs-2023-0043 · Cited by 18
“Besides the work of Cohen et al. (2006) from which the data origin, they were used in other studies (Matwin et al. , 2010; Jonnalagadda and Petitti, 2013; Khabsa et al. , 2016; Ji et al. , 2017).”
BackgroundMethodPang, X., Saif‐Ur‐Rahman, K., Berhane, S., Yao, X., Kothari, K. U., Taneri, P. E., Thomas, J., & Devane, D. (2025) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2025.111738 · Cited by 13
Norman, C. R., Leeflang, M., & Aurélie Névéol, A. (2018) · Europe PMC (PubMed Central) · doi:10.5281/zenodo.2574709 · Cited by 13
Scells, H., Zuccon, G., & Koopman, B. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-45439-5_27 · Cited by 13
“While current practice prescribes all documents must be screened (therefore explicit assessments could be used for active learning), an initially poor ranking would require many assessments before the system is able to identify relevant documents.”
Cohen, A. M., Schneider, J., Fu, Y., McDonagh, M. S., Das, P., Holt, A. W., & Smalheiser, N. R. (2021) · medRxiv · doi:10.1101/2021.07.13.21260468 · Cited by 13
Self-citationSystematic review using a spiral approach with machine learning
Saeidmehr, A., Steel, P., & Samavati, F. F. (2024) · Systematic Reviews · doi:10.1186/s13643-023-02421-z · Cited by 17
“Second, we are interested in relevant references found (RRF) [29], that is how accurate a machine learning option becomes after a given percentage of screened articles.”
BackgroundTransformative Automation: AI in Scientific Literature Reviews
Zala, K., Acharya, B., Mashru, M., Palaniappan, D., Gerogiannis, V. C., Kanavos, A., & Karamitsos, I. (2024) · International Journal of Advanced Computer Science and Applications · doi:10.14569/ijacsa.2024.01501122 · Cited by 17
“The initial application of AI methods to automate SLR tasks occurred in 2006 [12], where it was suggested that neural networks could be used to automate the selection of relevant articles.”
MethodBERT for Complex Systematic Review Screening to Support the Future of Medical Research
Hasny, M., Vasile, A.-P., Gianni, M., Bannach‐Brown, A., Nasser, M., Mackay, M., donovan, D., Šorli, J., Domocos, I., Dulloo, M., et al. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-34344-5_21 · Cited by 12
Semantic Enrichment for Recommendation of Primary Studies in a Systematic Literature Review
Rizzo, G. F., Tomassetti, F., Vetrò, A., Ardito, L., Torchiano, M., Morisio, M., & Raphaël Troncy, A. (2015) · Digital Scholarship in the Humanities · doi:10.1093/llc/fqv031 · Cited by 12
“According to (Cohen et al., 2006), we decided to aim at a recall of 95%.”
BackgroundResultInfluentialPetersen, H., Poon, J., Poon, S. K.-S., & Loy, C. T. (2014) · JMIR Medical Informatics · doi:10.2196/medinform.3037 · Cited by 12
“Prior work addressing this issue include the modified voting perceptron method of Cohen et al [24], the factorized Complement Naïve Bayes model of Matwin et al [25], and support vector machine based approaches by Cohen et al [6,26] and Wallace et al [7,27] Attention has also been directed towards…”
slr-kit: A semi-supervised machine learning framework for systematic literature reviews
Facchinetti, T., Benetti, G., Giuffrida, D., & Nocera, A. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109266 · Cited by 12
Mitrov, G., Stanoev, B., Gievska, S., Mirceva, G., & Zdravevski, E. (2024) · Big Data and Cognitive Computing · doi:10.3390/bdcc8090110 · Cited by 17
Ofori-Boateng, R., Escobar, T. G. T., Aceves‐Martins, M., Wiratunga, N., & Moreno‐García, C. F. (2024) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2024.102989 · Cited by 11
“On the other hand, the three public datasets included the Angiotensin-converting-enzyme (ACE) Inhibitors dataset authored by Cohen et al. [13], the van de Schoot_2017 dataset [22] focusing on post-traumatic stress disorder (PTSD), and the Kwok dataset [43] discussing Virus Metagenomics in animals.”
BackgroundMethodInfluentialSeed-Driven Document Ranking for Systematic Reviews: A Reproducibility Study
Wang, S., Scells, H., Mourad, A., & Zuccon, G. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_46 · Cited by 11
Appraising the Potential Uses and Harms of LLMs for Medical Systematic Reviews
Yun, H., Marshall, I. J., Trikalinos, T. A., & Wallace, B. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-main.626 · Cited by 23
Scientific Argumentation Detection as Limited-domain Intention Recognition.
Teufel, S. (2014) · Cited by 12
Sousa, M. S. A., Peiris, S., Figueiró, M. F., Haby, M. M., Baraldi, A. C., Revéiz, L., & Souza, J. P. (2026) · Systematic Reviews · doi:10.1186/s13643-025-02842-y · Cited by 13
Metadata
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
