The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
The cited source says what the claim attributes to it.
In the citing paper
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
The cited source says what the claim attributes to it.
In the citing paper
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 622
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- 2005, BMC bioinformatics [electronic resource], 6, S16
- 1980, PROGRAM, 14, 127
Cited by
Sorted by citations
Items 401–411 of 411
Automated Study Selection Using Representation Learning and Clustering
Owolabi, F. (2026) · Lecture notes in networks and systems · doi:10.1007/978-3-032-20897-2_53
Descamps, J., Bouguennec, N., Porcher, R., Resche‐Rigon, M., & Bouché, P.-A. (2026) · Orthopaedics & Traumatology Surgery & Research · doi:10.1016/j.otsr.2026.104793
Descamps, J., Bouguennec, N., Porcher, R., Resche‐Rigon, M., & Bouché, P.-A. (2026) · Revue de Chirurgie Orthopédique et Traumatologique · doi:10.1016/j.rcot.2026.07.013
Spanodimitriou, Y., Ejaz, K. T., Ciampi, G., Scorpio, M., Masullo, M., Rosato, A., Maffei, L., & Sibilio, S. (2026) · Sensors · doi:10.3390/s26175640
Large language models for literature screening in conceptually complex and interdisciplinary reviews
Cheng, N., Dong, H., Han, X., Tan, B., & Zhu, S. (2026) · Aslib Journal of Information Management · doi:10.1108/ajim-06-2025-0363
Screen: browser-only title–abstract screening with active learning — reproducible benchmark
Ahmad, M. (2026) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.23132679
Label-free, low-cost AI ranking for systematic review screeningPreprint
Jo, T., 이동현, & Lee, L. (2026) · medRxiv · doi:10.64898/2026.10.09.26365173
Nunnery, J. A. (2026) · Journal of Higher Education Theory and Practice · doi:10.33423/5tg2ts54
“Automated citation classification to cut screening workload dates back at least two decades (Cohen, Hersh, Peterson, & Yen, 2006).”
Improving Classification and Attribute Clustering: An Iterative Semi-supervised Approach
Seifi, F. (2015) · uO Research (University of Ottawa) · doi:10.20381/ruor-2821
An effective general purpose approach for automated biomedical document classification.
(2006) · Europe PMC (PubMed Central) · Cited by 44
Self-citationImproving Active Learning in Systematic Reviews
Singh, G., Thomas, J., & Shawe-Taylor, J. (2018) · arXiv · doi:10.48550/arxiv.1801.09496 · Cited by 11
“Active learning is an iterative process that starts from a small set of labelled studies and gradually learns to differentiate between relevant and irrelevant studies [10, 15, 4].”
BackgroundMethod
Metadata
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
