The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
The cited source says what the claim attributes to it.
In the citing paper
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
The cited source says what the claim attributes to it.
In the citing paper
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 622
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- 2005, BMC bioinformatics [electronic resource], 6, S16
- 1980, PROGRAM, 14, 127
Cited by
Sorted by citations
Items 301–400 of 411
Lagopoulos, A., & Tsoumakas, G. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.09752 · Cited by 1
“Early studies were focused on classifying documents as relevant or not to a review topic (Cohen et al., 2006) and dealing with imbalanced datasets (Cohen, 2006).”
Machine Learning in Evidence Synthesis Research
Carrasco‐Labra, A., Urquhart, O., & Spallek, H. (2021) · Springer eBooks · doi:10.1007/978-3-030-71881-7_12 · Cited by 1
Schaut, W., Shrivastav, A., Ramakrishnan, S., & Bowden, R. (2023) · Frontiers in Medicine · doi:10.3389/fmed.2023.1072767 · Cited by 2
“The achieved reduction in manual labor (ranging from 84 to 90%) is consistent with other reported increases in efficiency of systematic literature reviews using ML algorithms and text mining, and reduces the burden on human screeners (12, 13, 16).”
BackgroundResultAutomated SLR with a Few Labeled Papers and a Fair Workload Metric
Faria, A. V. A., de Melo, M. K., de Oliveira, F. A. R., Weigang, L., & Celestino, V. R. R. (2023) · Lecture notes in business information processing · doi:10.1007/978-3-031-43088-6_1 · Cited by 1
Lee, E. W., Xiong, B., Yang, C., & Ho, J. C. (2024) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3627673.3679940 · Cited by 1
“Between the original Poincaré embedding model and HypMix, the results show that HypMix significantly outperforms the former and highlights the effectiveness of the components that we propose.”
eCommerce Search and Discovery
Kallumadi, S. T., & King, T. H. (2024) · ACM eBooks · doi:10.1145/3674127.3674140 · Cited by 1
Active Learning for Medical Article Classification with Bag of Words and Bag of Concepts Embeddings
Pytlak, R., Cichosz, P., Fajdek, B., & Jastrzębski, B. (2025) · Applied Sciences · doi:10.3390/app15147955 · Cited by 1
Semantically Orthogonal Framework for Citation Classification: Disentangling Intent and Content
Duan, C., & Tan, Z. (2026) · Lecture Notes in Computer Science · doi:10.1007/978-3-032-05409-8_12 · Cited by 3
“…understanding provides the technological backbone for a new generation of digital libraries and intelligent research assistants, enabling advanced services such as automated literature reviews [3,15,27,28], the detection of scientific controversies [41], and impact-aware scientific search [35,63].”
Using Artificial Intelligence for Text Screening in a Systematic Review of Cardiotoxicity
Canfield, S. E., Aziz, M. K., Omar, M. I., N'Dow, J. M., Schijvenaars, B. J. A., Ghith, J., Ahmed, K. R., Molony, D. A., & Stenzl, A. (2025) · European Urology Open Science · doi:10.1016/j.euros.2025.09.005 · Cited by 1
“For evaluating the efficiency of both these ranking meth-ods, work saved over sampling (WSS) [20,31] was used, which measures the work saved at a certain level of recall (ie, proportion of relevant records retrieved) [32].”
Simulation-based active learning for systematic reviews: A scoping review of literature
Teijema, J. J., Ribeiro, G., Seuren, S., Anadria, D., Bagheri, A., & van de Schoot, R. (2025) · Journal of Information Science · doi:10.1177/01655515251379058 · Cited by 1
ASReview Dory: Bringing new and exciting models to ASReview LAB
van der Kuil, T., Teijema, J. J., Bruin, J. D., & van de Schoot, R. (2026) · Software Impacts · doi:10.1016/j.simpa.2025.100809 · Cited by 1
“…learning has been shown to be an effective strategy to reduce this workload by continuously prioritising the most relevant records for screening [2], and the rapid development of machine learning and natural language The code (and data) in this article has been certified as Reproducible by Code…”
Zero-shot interpretable biomedical literature appraisal with generative large language models
Zhou, F., Afzal, M., Saha, A., Parrish, R., Haynes, R. B., Iorio, A., & Lokker, C. (2026) · JAMIA Open · doi:10.1093/jamiaopen/ooag043 · Cited by 1
“43 MCC was used as the main metric to establish relative performance as it is a balanced measure that is particularly useful for imbalanced classification tasks when the cost of false positives and false negatives is equally important.”
Wang, X., Dadashipour, F., Basori, Maeda, Y., & Richardson, J. C. (2026) · Educational Technology Research and Development · doi:10.1007/s11423-026-10629-8 · Cited by 1
“A key development in 2006 demonstrated that machine learning (ML) classifiers could reduce screening burden while maintaining strong coverage of relevant studies (Cohen et al., 2006).”
Sušnjak, T., Palffy, C., Zimina, T., Altynbekova, N., Garg, K., & Gilbert, L. (2026) · Scientometrics · doi:10.1007/s11192-026-05681-3 · Cited by 1
“…Ixodes, Erythema, migrans, tick-borne, tickborne, tick borne Prior work supports the use of machine learning for medical text classification (Cohen et al., 2006), and recent LLM-based workflows have shown competitive performance on related screening and extraction tasks (Li et al., 2025c;…”
Nunnery, J. A. (2026) · Research Synthesis Methods · doi:10.1017/rsm.2026.10116 · Cited by 1
Learning with Augmented Multi-Instance View
Zhu, Y., Wu, J., Jiang, Y., & Zhou, Z. (2014) · Cited by 1
Natural Language Processing and Machine Learning Techniques Help Achieve a Better Medical Practice
Frunza, O., & Inkpen, D. Z. (2012) · Advances in medical technologies and clinical practice book series · doi:10.4018/978-1-4666-1803-9.ch016 · Cited by 1
Reducing Workload of Systematic Review Searching and Screening Processes
Scells, H. (2017) · Electronic workshops in computing · doi:10.14236/ewic/fdia2017.2 · Cited by 1
Hersh, W. (2020) · Health informatics · doi:10.1007/978-3-030-47686-1_8 · Cited by 1
Self-citationNLP Applications—Biomedical Literature
Zhang, R., Kastrin, A., Hristovski, D., Marcelo Fiszman, B., & Halil Kilicoglu, B. (2024) · Cognitive informatics in biomedicine and healthcare · doi:10.1007/978-3-031-55865-8_13 · Cited by 1
Artificial intelligence applied in human health technology assessment: a scoping review protocol
Komoda, D. S., Cardoso, M. M. D. A., Fernandes, B. D., Visacri, M. B., & Corrêa, C. R. S. (2024) · JBI Evidence Synthesis · doi:10.11124/jbies-23-00377 · Cited by 1
Galuščáková, P., Oard, D. W., & Nair, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674136 · Cited by 2
Gómez, E., Cuesta, H., Gkiokas, A., Gómez-Cañón, J. S., Porcaro, L., & Yesiler, F. (2024) · ACM eBooks · doi:10.1145/3674127.3702959 · Cited by 1
Baeza-Yates, R. A., Murgai, L., & Dudy, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674138 · Cited by 1
Molléri, J. S., Benitti, F. B. V., & Fernandes, A. M. D. R. (2013) · Anais SULCOMP · Cited by 1
Machine learning in health informatics: making better use of domain experts
Brodley, C. E., & Wallace, B. (2012) · Cited by 1
Active Learning for the Automation of Medical Systematic Review Creation
El-Gayar, O., Liu, J., & Timsina, P. (2015) · Journal of the Association for Information Systems · Cited by 1
Systematic Review using a Spiral approach with Machine LearningPreprint
Saeidmehr, A., Steel, P., & Samavati, F. F. (2023) · Research Square · doi:10.21203/rs.3.rs-2497596/v1 · Cited by 3
PharmD, L. T., Trampoukis, C., & Polymenakos, K. (2025) · medRxiv · doi:10.1101/2025.07.16.25331632 · Cited by 1
AIM review tool: artificial intelligence for smarter systematic review screening
Mena, S., Rituerto-González, E., Coutts, F., von Trott, J., Jacobs, G. R., Bryant, L., Moles, L., Sirbu, N., Promet, L., Oliver, D., et al. (2026) · npj Artificial Intelligence · doi:10.1038/s44387-026-00080-8
Rahgozar, A., & Mortezaagha, P. (2026) · arXiv · doi:10.48550/arxiv.2608.14551
“• WSS@95 : Work Saved over Sampling at 95% recall, defined following Cohen et al. [2006] as WSS@ R = ( TN + FN ) /N − (1 − R ) — the fraction of records the reviewer can skip (relative to random screening) while still recovering 95% of true includes.”
InfluentialPitre, T., Zeraatkar, D., Granton, J., Busse, J., Rochwerg, B., & Guyatt, G. (2026) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2026.112514 · Cited by 2
Personalized Medicine through Automatic Extraction of Information from Medical Texts
Frunza, O. (2012) · uO Research (University of Ottawa) · doi:10.20381/ruor-5599
“In (Cohen et al., 2006), the WSS measure is applied to report the reduction in reviewer’s work when retrieving 95% of the relevant documents, but the precision was very low.”
BackgroundMethodInfluential“entertain me” Supporting Complex Search Tasks
Belkin, N. J., Clarke, C. L. A., Gao, N., Kamps, J., Jussi Karlgren, C., Belkin, N., Clarke, C., Gao, N., Kamps, J., Jussi Karlgren, C., et al. (2011)
Extracting Knowledge Claims for Automatic Evidence Synthesis Using Semantic Technology
Guo, J. (2017) · Lecture notes in computer science · doi:10.1007/978-3-319-58694-6_37
A general method for regularizing tensor decomposition methods via pseudo-dataPreprint
Gottesman, O., Pan, W., & Doshi‐Velez, F. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1905.10424
“) Cohen et al. [2013], Duchi et al.”
Semi-Automating Knowledge Base Construction for Cancer GeneticsPreprint
Wadhwa, S., Yin, K., Hughes, K. S., & Wallace, B. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.08146
“And there has been some work specifically aiming to aid medical evidence synthesis via NLP technologies [Lehman et al., 2019, Marshall et al., 2018, Cohen et al., 2006, Brockmeier et al., 2019, Schmidt et al., 2020].”
BackgroundConducting a Systematic Review: Trends in Machine Learning and Text Mining
Falco, M., & Berdiñas, I. (2020) · El Servicio de Difusión de la Creación Intelectual (National University of La Plata)
Seed-driven document ranking for systematic reviews in evidence-based medicine
Lee, E. (2020) · DR-NTU (Nanyang Technological University) · doi:10.32657/10356/146709
Sharma, N. K. (2022) · International Journal of Advanced Research in Science Communication and Technology · doi:10.48175/ijarsct-3885
Structural Complexity and Performance of Support Vector Machines
Olorisade, B. K., Pearl Brereton, A., & András, P. (2022) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn55064.2022.9892368
“In the case of applications of SVM to classification problems, it is often found that the resulting SVMs have a high number of support vectors [11], [13]–[15].”
BackgroundMethodInfluentialFeature selection analysis of systematic review citation screening algorithm
Li, S., Du, Y.-C., Yun, Y.-L., & Liu, H.-G. (2022) · IEEE Advanced Information Management, Communicates, Electronic and Automation Control Conference (IMCEC) · doi:10.1109/imcec55388.2022.10020072
Orel, E., Ciglenečki, I., Thiabaud, A., Temerev, A., Calmy, A., Keiser, O., & Merzouki, A. (2023) · medRxiv · doi:10.1101/2023.02.20.23286179
Lee, E. W., & Ho, J. C. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-28244-7_35
Terminology for the Description of the Diagnostic Studies in the Field of EBM
Grabar, N., Trinquart, L., & Colombet, I. (2011) · Studies in health technology and informatics · doi:10.3233/978-1-60750-806-9-769
“Evidence-Based Medicine; Review, Systematic; Language; Natural Language Processing; Terminology”
Addressing the Challenges of Reconstructing Systematic Reviews Datasets V2Preprint
Neeleman, R. C., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R. (2023) · doi:10.31234/osf.io/jfcbq
Lucini, F. R., Fogliatto, F. S., Calegari, R., da Silveira, G. J. C., neyeloff, J., Anzanello, M. J., Kuchenbecker, R. D. S., & Schaan, B. D. (2018) · Anais do Simpósio Brasileiro de Pesquisa Operacional · doi:10.59254/sbpo-2018-85299
Is Hospital Management Associated with Quality of Care? A Systematic Review of the Global EvidencePreprint
Ward, C., Phiri, E., Goodman, C. A., Mipando, A. L. N., Malata, M. P., Manda, W. C., Mwapasa, V., & Powell‐Jackson, T. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4671836
Citation Style Classification: a Comparison of Machine Learning Approaches
Kopan, A., Smirnova, A. V., Shchuckin, I., Makeev, V., & Chernishev, G. A. (2023) · IEEE ... International Conference on Data Mining workshops · doi:10.1109/icdmw60847.2023.00139
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674128
Searching Personal Collections
Bendersky, M., Metzler, D., Najork, M., & Wang, X. (2024) · ACM eBooks · doi:10.1145/3674127.3674142
Query-driven Search Functionality
Li, L. (2024) · ACM eBooks · doi:10.1145/3674127.3674133
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702962
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3674129
Alonso, O., & Baeza-Yates, R. (2024) · ACM eBooks · doi:10.1145/3674127.3702961
Hersh, W., & Roberts, K. E. (2024) · ACM eBooks · doi:10.1145/3674127.3702960
Self-citationBast, H., Kalmbach, J., Klumpp, T., & Korzen, C. (2024) · ACM eBooks · doi:10.1145/3674127.3674134
ASReview LAB v2: Open-Source Text Screening with Multiple Agents and OraclesPreprint
Bruin, J. D., Lombaers, P., Kaandorp, C., Teijema, J. J., van der Kuil, T., Yazan, B., Dong, A., & van de Schoot, R. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5136987
Sušnjak, T., Palffy, C., Zimina, T., Altynbekova, N., Garg, K., & Gilbert, L. K. (2025) · medRxiv · doi:10.1101/2025.04.03.25325216
“This approach is supported by literature, where machine learning has been successfully used to classify medical texts for some time [108].”
Buitrago‐García, D., Courvoisier, D. S., Capderou, S., Iudici, M., & Mongin, D. (2025) · doi:10.2196/preprints.77943
Optimizing the Research Process: The Role of AI Tools in Scientific Literature Searches
Mandić, A. (2025) · International Convention on Information and Communication Technology, Electronics and Microelectronics · doi:10.1109/mipro65660.2025.11131993
Teperikidis, E., Trampoukis, C., & Polymenakos, K. (2025) · Journal of Cardiovascular Pharmacology · doi:10.1097/fjc.0000000000001768 · Cited by 2
Ding, Y., Jiang, X., & Akinseloyin, O. (2025) · medRxiv · doi:10.1101/2025.11.08.25339805
“Since Cohen’s ground-breaking work on machine learning for automated abstract screening in 2006 [9], two decades of research and development have resulted in a lot of computer tools for technology-assistance evidence synthesis.”
Towards AI-supported SLR Automation: Citation Screening with TransformersPreprint
Tsakpinis, A., & Wolff, C. G. (2025) · SSRN Electronic Journal · doi:10.2139/ssrn.5760724
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374721
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374727
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374724
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · Figshare · doi:10.6084/m9.figshare.13100809
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374733
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374745
Natukunda, A., & Muchene, L. K. (2024) · Figshare · doi:10.6084/m9.figshare.26558960
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · Figshare · doi:10.6084/m9.figshare.13100812
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374739
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374730
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Figshare · doi:10.6084/m9.figshare.14361540
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374736
Nama, N., Hennawy, M., Barrowman, N., O'Hearn, K., Sampson, M., & McNally, J. D. (2021) · Figshare · doi:10.6084/m9.figshare.14374742
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · Figshare · doi:10.6084/m9.figshare.13100815
Zhang, Y., Liang, S., Feng, Y., Wang, Q., Sun, F., Chen, S., Yang, Y., He, X., Zhu, H., & Pan, H. (2022) · Figshare · doi:10.6084/m9.figshare.18462027
Natukunda, A., & Muchene, L. K. (2024) · London School of Hygiene & Tropical Medicine · doi:10.6084/m9.figshare.26558963
Lee, T.-F., Hsieh, Y.-W., Yang, P.-Y., Tseng, C.-H., Lee, S.-H., Yang, J., Chang, L., Wu, J., Tseng, C.-D., & Chao, P.-J. (2024) · Figshare · doi:10.6084/m9.figshare.24971343
Neeleman, R. C., Leenaars, C. H. C., Oud, M., Weijdema, F., & van de Schoot, R. (2024) · Figshare · doi:10.6084/m9.figshare.26682223
Active Learning Models for Systematic Literature Review: A Railway Drainage Case
Gorbunov, O., Sinsabvarodom, C., & Rinchumphu, D. (2025) · International Joint Symposium on Artificial Intelligence and Natural Language Processing (iSAI-NLP) · doi:10.1109/isai-nlp66160.2025.11320463
“[6], [9] determines the percentage of non-relevant documents that the classifier successfully screens out while finding 95% of the relevant documents.”
InfluentialAI-powered analysis of bronchiectasis research trends: methodological considerations
Yu, Z. (2026) · European Respiratory Journal · doi:10.1183/13993003.01843-2025
Efficient Citation Screening by Weak Classifier Ensemble*
Jiang, X., Akinseloyin, O., & Palade, V. (2025) · ACM/IEEE Joint Conference on Digital Libraries · doi:10.1109/jcdl67857.2025.00042
“In the screening prioritization setting , the voter generates a new score to rank the documents and performance metrics are MAP (Mean Average Precision), WSS95 and WSS100 (WSS at 95% and 100% recall of include studies) [13], and R@ k % (recall at top k %, k = 5, 10, 30, 50).”
Hao, Z., Duan, Y., Wang, Y., & Qian, H. (2025) · doi:10.2196/preprints.81833
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2026) · SSRN Electronic Journal · doi:10.2139/ssrn.6263105
Rostam, Z. R. K., & Kertész, G. (2026) · IEEE World Symposium on Applied Machine Intelligence and Informatics (SAMI) · doi:10.1109/sami68106.2026.11420295
“Cohen et al. [17] propose a voting perceptron-based system for classifying citations in systematic drug class reviews.”
Haroutunian, L. (2022) · theses.fr (ABES) · doi:10.70675/a3de09d0z30faz4d04z846fz1d75d3f69a3a
Chen, X., Meng, X., Lv, S., Zhang, Y., & Chen, Y. J. (2026) · Scientometrics · doi:10.1007/s11192-026-05594-1
“This research, including citation intent identification (Chang 2013; Galgani et al. 2015; Cohen et al. 2006) and citation sentiment analysis (Athar and Teufel 2012; Hernández and Gómez 2015; Yousif et al. 2019), elevates the granularity of evaluation from macroscopic statistics to microscopic…”
Ahn, S., Kim, S., & Valdes, J. (2026) · Springer international handbooks of education · doi:10.1007/978-3-031-88898-4_24-1
Buitrago‐García, D., Courvoisier, D. S., Capderou, S., Iudici, M., & Mongin, D. (2026) · JMIR Medical Informatics · doi:10.2196/77943
“This approach has demonstrated good and reproducible performance [7] but it lacks versatility and requires task-specific training.”
Concerns of Using Large Language Models in Health Care Research and Practice: Umbrella Review
Yarar, F., Addis, P., Fairweather, M., Craig, D., & O’Keefe, H. (2026) · Journal of Medical Internet Research · doi:10.2196/87804 · Cited by 1
“However, the conversation and movement toward automation in this field have been ongoing for 2 decades with little to no progress [21-23].”
Jóźwiak, Á. B., Imre, A., Józwiak‐Hagymásy, J., Tittmann, J., Nagy, Á., Kovács, S., Kardas, P., van Boven, J. F. M., Mommers, I., & Ágh, T. (2026) · Expert Review of Pharmacoeconomics & Outcomes Research · doi:10.1080/14737167.2026.2691995
“…learning and text mining methods, which were shown to reduce screening workloads by approximately 30–70% in some settings, though often at the cost of reduced recall and with substantial requirements for labeled training data, technical expertise, and review-specific model retraining [11–13].”
How Can AI Augment Access to Justice? Public Defenders' Perspectives on Responsible AI Adoption
Cheong, I., Liu, P., Stammbach, D., & Henderson, P. (2026) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3805689.3806526 · Cited by 3
Enhancing Title and Abstract Priority Screening Through EdRoSi AI PipelinePreprint
Lecot, P., Tonoli-Catez, H., Noseda, A., Buisse, T., Chanut, S., & Chanel, I. (2026) · medRxiv · doi:10.64898/2026.06.26.26356718
Metadata
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
