The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
The cited source says what the claim attributes to it.
In the citing paper
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
The cited source says what the claim attributes to it.
In the citing paper
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 622
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- 2005, BMC bioinformatics [electronic resource], 6, S16
- 1980, PROGRAM, 14, 127
Cited by
Sorted by citations
Items 201–300 of 411
Large-scale simulation study of active learning models for systematic reviews
Teijema, J. J., Bruin, J. D., Bagheri, A., & van de Schoot, R. (2025) · International Journal of Data Science and Analytics · doi:10.1007/s41060-025-00777-0 · Cited by 12
“Even the high-quality Cohen dataset [11], often considered the gold standard in the field, is limited to drug class reviews.”
InfluentialHelmick, H. D. B., Nanda, G., Ettestad, S., Liceaga, A. M., & Kokini, J. L. (2021) · Journal of Food Science · doi:10.1111/1750-3841.15940 · Cited by 10
Ofori-Boateng, R., Aceves‐Martins, M., Jayne, C., Wiratunga, N., & Moreno‐García, C. F. (2023) · Procedia Computer Science · doi:10.1016/j.procs.2023.08.149 · Cited by 14
Using neural networks to support high-quality evidence mapping
Røst, T. B., Slaughter, L. A., Øystein Nytrø, A., Muller, A. E., & Vist, G. E. (2021) · BMC Bioinformatics · doi:10.1186/s12859-021-04396-x · Cited by 10
Performance of SVM and Bayesian classifiers on the systematic review classification task
Matwin, S., Kouznetsov, A., Inkpen, D. Z., Frunza, O., & O’Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.009555 · Cited by 10
Improving Endpoint Detection to Support Automated Systematic Reviews.
(2016) · PubMed Central · Cited by 10
Castells, P., & Jannach, D. (2024) · ACM eBooks · doi:10.1145/3674127.3674131 · Cited by 10
ContReviews: A content-based recommendation system for updating Living Evidences in health care
Tenti, P., Thomas, J., Peñaloza, R., & Pasi, G. (2025) · Knowledge-Based Systems · doi:10.1016/j.knosys.2025.112981 · Cited by 10
Training sample selection: Impact on screening automation in diagnostic test accuracy reviews
van Altena, A. J., Spijker, R., Leeflang, M., & Olabarriaga, S. D. (2021) · Research Synthesis Methods · doi:10.1002/jrsm.1518 · Cited by 9
“This concept of performance is captured in the metric ‘Work Saved over Sampling’ (WSS), introduced by Cohen et al..(8) For a specified level of recall,‡ WSS measures the fraction of documents that a review author does not need to read as a result of the ranking, as compared to a random ordering.”
BackgroundSystematic literature review (SLR) automation: a systematic literature review
Hamad, Z., & Salim, N. B. (2014) · Journal of Theoretical and Applied Information Technology · Cited by 9
Active Literature Discovery for Scoping Evidence Reviews How Many Needles are There
Wallace, B., Dahabreh, I. J., Moran, K. H., Brodley, C. E., & Trikalinos, T. A. (2013) · Cited by 9
A Study of the Influence of Textual Features in Learning Medical Prior Authorization
Magalhães, G. V., Vieira, J. P. A., Santos, R. L. D. S., Barbosa, J. L. N., Neto, P. D. A. D. S., & Moura, R. S. (2019) · Proceedings - IEEE Symposium on Computer-Based Medical Systems · doi:10.1109/cbms.2019.00021 · Cited by 9
Thayer, K. A., Angrish, M. M., Arzuaga, X., Carlson, L. M., Davis, A., Dishaw, L. V., Druwe, I. L., Gibbons, C. F., Glenn, B. S., Jones, R., et al. (2022) · Environment International · doi:10.1016/j.envint.2022.107468 · Cited by 11
Uthman, O. A., Court, R. A., Enderby, J., Al-Khudairy, L., Nduka, C. U., Mistry, H., Meléndez‐Torres, G. J., Taylor‐Phillips, S., & Clarke, A. (2022) · Health Technology Assessment · doi:10.3310/udir6682 · Cited by 11
“38 We defined work saved as the percentage of papers that meet the original search criteria but are not required to be read by the reviewers (because they have been screened out by the classifier).”
BackgroundMethodInfluentialHow to Read Less: Better Machine Assisted Reading Methods for Systematic Literature Reviews.Preprint
Yu, Z., Kraft, N. A., & Menzies, T. (2016) · arXiv (Cornell University) · Cited by 9
Using Chao’s Estimator as a Stopping Criterion for Technology-Assisted Review
Bron, M. P., van der Heijden, P. G. M., Feelders, A., & Siebes, A. P. J. M. (2025) · ACM Transactions on Information Systems · doi:10.1145/3724116 · Cited by 8
“5 with a target recall of 95 %.”
Zhou, F., Parrish, R., Afzal, M., Saha, A., Haynes, R. B., Iorio, A., & Lokker, C. (2025) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2025.104825 · Cited by 9
Reducing the user labeling effort in effective high recall tasks by fine-tuning active learning
Bianco, G. D., Duarte, D., & Gonçalves, M. A. (2023) · Journal of Intelligent Information Systems · doi:10.1007/s10844-022-00772-y · Cited by 8
“Several works address HIRE as supervised classification problem since the documents are explicitly identified as “relevant” or “non-relevant” (Cohen et al. 2006).”
BackgroundZamantakis, A., Merle, J. L., Queiroz, A. A. F. L. N., Zapata, J. P., Deskins, J. S., Pachicano, A. M., Mongrella, M. C., Li, D. H., Benbow, N., Gallo, C. G., et al. (2024) · Implementation Science Communications · doi:10.1186/s43058-024-00638-0 · Cited by 8
Modernizing Evidence Synthesis for Evidence-Based Medicine
Wallace, B., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2014) · Elsevier eBooks · doi:10.1016/b978-0-12-398476-0.00012-9 · Cited by 8
The screening phase in systematic reviews: Can we speed up the process?
Rožanc, I., & Mernik, M. (2021) · Advances in computers · doi:10.1016/bs.adcom.2021.01.006 · Cited by 8
Large-Scale Simulation Study of Active Learning Models for Systematic ReviewsPreprint
Teijema, J. J., Bruin, J. D., Bagheri, A., & van de Schoot, R. (2023) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/2w3rm · Cited by 7
Christensen, K. L. Y., Bommarito, P. A., Shaffer, R. M., Davis, B. C., & Lee, A. L. (2025) · Journal of Toxicology and Environmental Health Part B · doi:10.1080/10937404.2025.2559344 · Cited by 7
Outcome-based Evaluation of Systematic Review Automation
Kusa, W., Zuccon, G., Knoth, P., & Hanbury, A. (2023) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3578337.3605135 · Cited by 13
“We follow the evaluation procedure of CLEF TAR and calculate the following traditional evaluation measures: Mean Average Precision (MAP ), last relevant found, Recall@k% of top-ranked publications, with k in [5, 10, 20, 30, 50], Work Saved over Sampling at r% of recall with r in [95%, 100%] (WSS@95%,WSS@100%), nDCG@20…”
BackgroundMethodInfluentialByrne, F., Hofstee, L., Teijema, J. J., Bruin, J. D., & van de Schoot, R. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02587-0 · Cited by 7
“It enables the user to theoretically find relevant papers by screening only a fraction of the most likely relevant records [9].”
CRUISE-Screening: Living Literature Reviews Toolbox
Kusa, W., Knoth, P., & Hanbury, A. (2023) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3583780.3614736 · Cited by 9
“The output of the literature review can be exported in a json format 6 .”
BackgroundMethodRomanov, S., Siqueira, A. S., Bruin, J. D., Teijema, J. J., Hofstee, L., & van de Schoot, R. (2024) · Data Intelligence · doi:10.1162/dint_a_00244 · Cited by 6
“…systematic review implies marking scientific papers as relevant or irrelevant “by hand”, the process of labeling can be optimized with the use of machine learning (ML) – especially pipelines based on active learning [1-3], a constant interaction between a human and a machine learning model.”
BackgroundA Dataset of Systematic Review Updates
Alharbi, A., & Stevenson, M. (2019) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3331184.3331358 · Cited by 7
Limsi@Clef Ehealth 2017 Task 2: Logistic Regression For Automatic Article Ranking
Norman, C. R., Leeflang, M., & Aurélie Névéol, A. (2017) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.887618 · Cited by 7
Wiwatthanasetthakarn, P., Ponthongmak, W., Looareesuwan, P., Tansawet, A., Numthavaj, P., McKay, G. J., Attia, J., & Thakkinstian, A. (2024) · Journal of Medical Internet Research · doi:10.2196/56863 · Cited by 6
“For each experimental scenario, the number needed to screen (NNS) and %Reduced workload, also known as work, saved oversampling [37], were estimated using the following formulae: where TP and FP are true and false positive studies (predicted eligible studies), and N is the total number of studies…”
Accelerating the Early Identification of Relevant Studies in Title and Abstract Screening
Bravo, Á., Bennetts, L. E., & Atanasov, P. (2021) · International Symposium on Computer Science and Intelligent Control (ISCSIC) · doi:10.1109/iscsic54682.2021.00034 · Cited by 6
Efficient evidence selection for systematic reviews in traditional Chinese medicine
Li, Y., Huang, Z., Luan, Z., Xu, S., Zhang, Y., Wu, L., Wu, D., Han, D., & Liu, Y. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02430-z · Cited by 7
Towards a multi-level framework for supporting systematic review — A pilot study
Li, D., Wang, Z., Shen, F., Murad, M. H., & Liu, H. (2014) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm.2014.6999266 · Cited by 6
“In 2006, ting for citation eparations [19].”
MethodCai, T. A., He, Z., Hong, C., Zhang, Y., Ho, Y.-L. A., Honerlaw, J. P., Geva, A., Panickan, V. A., King, A., GAGNON, D. R., et al. (2022) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2022.104109 · Cited by 6
Hassler, E. E., Hale, D. P., & Hale, J. E. (2018) · Information and Software Technology · doi:10.1016/j.infsof.2018.02.001 · Cited by 6
Systematic review automation methods
Norman, C. R. (2020) · theses.fr (ABES) · doi:10.70675/044ffa34z5c9bz433az99c8z64697dcd28a2 · Cited by 5
Spiero, I., Leeuwenberg, A., Moons, K. G. M., Hooft, L., & Damen, J. A. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10025 · Cited by 5
“3 After one of the first attempts to accelerate the screening process with automated classification of titles and abstracts by Cohen et al. 4 many artificial intelligence (AI)-based (semi-)automated screening tools and algorithms have been developed.”
König, L., Zitzmann, S., & Hecht, M. (2025) · Research Synthesis Methods · doi:10.1017/rsm.2025.10053 · Cited by 5
“4,18,24 These tools typically employ an active learning approach—a human-in-the-loop method in which an ML algorithm determines the screening order and continuously updates its predictions based on the decisions of a human reviewer (see Figure 1).”
Rich, A. J., McGorray, E. L., Baldwin-SoRelle, C., Cawley, M. A., Grigg, K., Beach, L. B., Phillips, G., & Poteat, T. (2025) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2025.1860 · Cited by 5
“However, despite these tools’ potential, [12–18], they have not often been leveraged to streamline the process of conducting systematic reviews [11].”
Lee, E. W., Wallace, B., Galaviz, K. I., & Ho, J. C. (2020) · ACM Conference on Health, Inference, and Learning (CHIL) · doi:10.1145/3368555.3384463 · Cited by 5
“Our general strategy builds upon a body of work on semi-automating citation screening for evidence syntheses via machine learning [15, 41, 49, 52].”
BackgroundMethodInfluentialExploitation of semantic similarity for adaptation of existing terminologies within biomedical area
Dupuch, M., Trinquart, L., Colombet, I., Jaulent, M., & Grabar, N. (2010) · Cited by 5
Artificial Intelligence in Evidence-Based Medicine
Nowak, A. J. (2021) · Springer eBooks · doi:10.1007/978-3-030-58080-3_43-1 · Cited by 5
A Method to Automatically Identify the Results from Journal Articles
Gabb, H. A., Lučić, A., & Blake, C. L. (2015) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign) · Cited by 5
Silva, G., Neto, P. D. A. D. S., Moura, R. S., Araujo, A. C., Castro, O. C. C., & Ibiapina, I. M. D. S. (2019) · Brazilian Conference on Intelligent Systems · doi:10.1109/bracis.2019.00147 · Cited by 5
“This value is used in other studies as discussed in [4], [14], [15], since the recall of 1.”
MethodLLM-Guided Weak Supervision Improves Active Learning for Abstract ScreeningPreprint
Akinseloyin, O., Jiang, X., & Palade, V. (2025) · medRxiv · doi:10.1101/2025.08.24.25334314 · Cited by 6
“In the field of automated systematic review, the dominant performance metric for abstract screening is Work Saved over Sampling (Cohen et al., 2006), which is defined below.”
InfluentialEastaugh, C., Still, M., Beyer, F. R., Wallace, S. A., & O’Keefe, H. (2025) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70049 · Cited by 7
“However, the discussion regarding AI automation for systematic reviews has been ongoing for two decades [9, 10].”
Yao, X., Low, A., Sivajohanathan, D., Vella, E. T., Wang, P., Kaur, G., Saha, A., & SUSSMAN, J. S. (2025) · Intelligent Medicine · doi:10.1016/j.imed.2025.12.006 · Cited by 4
Repke, T., Tinsdeall, F., Danilenko, D., Graziosi, S., Müller-Hansen, F., Schmidt, L., Thomas, J., & van Valkenhoef, G. (2026) · Cochrane Evidence Synthesis and Methods · doi:10.1002/cesm.70068 · Cited by 6
“…TREC 2010, 2015, 2016 tracks on total recall and legal [57–59], the SYNERGY collection (reviews in psychology, medicine, biology, computer science, and maths) which extends the Cohen collection [60, 61], and CSMeD [62] which extends some of the previously mentioned collections and others [63, 64].”
Machine learning enables automated screening for systematic reviews and meta-analysis in urology
Menold, H. S., Wieland, V. L. S., Haney, C. M., Uysal, D., Wessels, F., Cacciamani, G. E., Michel, M. S., Seide, S., & Kowalewski, K. (2024) · World Journal of Urology · doi:10.1007/s00345-024-05078-y · Cited by 4
“WSS was introduced by Cohen et al. and is dedicated to be used for SRs [ 10 ].”
MethodSmooth Operators for Effective Systematic Review Queries
Scells, H., Schlatt, F., & Potthast, M. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591768 · Cited by 7
“Most research in this area focuses on methods that directly accelerate the screening phase; for example, active learning [8, 29, 55], text mining [3, 10, 31, 50, 52], and document ranking [20, 26, 54].”
BackgroundTrippas, J. R. (2024) · ACM eBooks · doi:10.1145/3674127.3674135 · Cited by 5
Bashir, R., Dunn, A. G., & Surian, D. (2020) · Research Synthesis Methods · doi:10.1002/jrsm.1473 · Cited by 4
Artificial Intelligence in Evidence-Based Medicine
Nowak, A. J. (2022) · Springer eBooks · doi:10.1007/978-3-030-64573-1_43 · Cited by 4
Using Machine Learning to Locate Evidence More Efficiently
Cawley, M. A. (2022) · Advances in library and information science (ALIS) book series · doi:10.4018/978-1-7998-9702-6.ch008 · Cited by 4
Li, X., Zhang, A., Al-Zaidy, R. A., Rao, A., Baral, S., Bao, L., & Giles, C. L. (2022) · PLoS ONE · doi:10.1371/journal.pone.0270034 · Cited by 6
“commonly used to evaluate ADC models that screen documents for systematic reviews [31], often called WSS@R, where R is the desired recall level.”
BackgroundVoMBaT: A Tool for Visualising Evaluation Measure Behaviour in High-Recall Search Tasks
Kusa, W., Lipani, A., Knoth, P., & Hanbury, A. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591802 · Cited by 6
“Another measure, Work Saved over Sampling ( 𝑊𝑆𝑆 ), measures the amount of work saved when using machine learning models to screen irrelevant publications [3].”
BackgroundMethodInfluentialSciMine: An Efficient Systematic Prioritization Model Based on Richer Semantic Information
Guo, F., Luo, Y., Yang, L., & Zhang, Y. (2023) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3539618.3591764 · Cited by 4
“Systematic review aims to use systematic and explicit methods to collect, identify and critically appraise relevant studies about one research theme [12, 54].”
MethodSharma, B., Willis, V. C., Huettner, C. S., Beaty, K. A., Snowdon, J. L., Xue, S., South, B. R., Jackson, G. P., Weeraratne, D., & Michelini, V. V. (2020) · JAMIA Open · doi:10.1093/jamiaopen/ooaa028 · Cited by 4
Why Did You Not Compare with That? Identifying Papers for Use as Baselines
Bedi, M., Pandey, T., Bhatia, S., & Chakraborty, T. (2022) · Lecture notes in computer science · doi:10.1007/978-3-030-99736-6_4 · Cited by 4
Mani, N. S., Hayes, B. E., Dodd, A., Yu, F., & Cawley, M. A. (2021) · Advances in Library and Information Science · doi:10.4018/978-1-7998-7258-0.ch017 · Cited by 4
Holland, C., Oakes, D. B., Sarna, M. K., Chai, K. E. K., Ng, L., & Moore, H. C. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02511-7 · Cited by 3
Leveraging Advanced Analytics Techniques for Medical Systematic Review Update
Timsina, P., El-Gayar, O., & Liu, J. (2015) · Hawaii International Conference on System Sciences (HICSS) · doi:10.1109/hicss.2015.121 · Cited by 4
“[2] used the perceptron algorithm to mine the title and abstract of an article and determine if the article should be included in the systematic review.”
InfluentialSchmitz, T., Bukowski, M., Koschmieder, S., Schmitz‐Rode, T., & Farkas, R. (2019) · Research Synthesis Methods · doi:10.1002/jrsm.1350 · Cited by 2
“To evaluate the performance of a text mining implementation the WSS score was introduced by Cohen et al. 16 The WSS score reflects the percentage of the workload saved compared with randomly sampling the dataset.”
MethodResultInfluentialModernizing evidence synthesis for evidence-based medicine
Saldanha, I. J., Adam, G. P., Schmid, C. H., Trikalinos, T. A., & Konnyu, K. J. (2023) · Elsevier eBooks · doi:10.1016/b978-0-323-91200-6.00006-1 · Cited by 3
Machine learning based system for the automation of systematic literature reviews
Pytlak, R., Bukhvalova, B. A., Cichosz, P., Fajdek, B., Grahek‐Ogden, D., Jastrzębski, B., Khan, M., Postupolski, J., Yazidi, A., & Waszkowski, R. (2023) · IEEE International Conference on Bioinformatics and Biomedicine (BIBM) · doi:10.1109/bibm58861.2023.10385372 · Cited by 3
“Moreover, to compare the results, we used the metric introduced in [40] which refers to the work saved as a result of the use of a semi–automated literature review.”
MethodNormalised Precision at Fixed Recall for Evaluating TAR
Kusa, W., Peikos, G., Staudinger, M., Lipani, A., & Hanbury, A. (2024) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3664190.3672532 · Cited by 4
Protocol-Driven Searches for Medical and Health-Sciences Systematic Reviews
Bouamrane, M.-M., Macdonald, C., Ounis, I., & Mair, F. S. (2011) · Lecture notes in computer science · doi:10.1007/978-3-642-23318-0_18 · Cited by 3
Reporting Statistical Validity and Model Complexity in Machine Learning based Computational Studies
Olorisade, B. K., Pearl Brereton, A., & András, P. (2017) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1145/3084226.3084283 · Cited by 3
“The datasets are part of the Drug Evaluation Review Program (DERP) reports made available through the collaboration between the Cochrane Centre and the Evidence based Practice Centres (EPC) of the Agency for Healthcare Research and Quality (AHRQ) [13].”
MethodInfluentialAutomating Document Discovery in the Systematic Review Process: How to Use Chaff to Extract Wheat
Norman, C. R., Leeflang, M., Zweigenbaum, P., & Aurélie Névéol, A. (2018) · HAL · doi:10.5281/zenodo.1173076 · Cited by 3
“We also report the WSS@95 scores (Cohen et al., 2006) in order to compare our results against the naive bayes methods of Matwin et al.”
BackgroundMethodResultInfluentialScientometrics: How to perform a Big Data Trend Analysis with ScienceMiner.
Frehe, V., Rugaitis, V., & Teuteberg, F. (2014) · GI-Jahrestagung · Cited by 3
What Does the Evidence Say? Models to Help Make Sense of the Biomedical Literature
Wallace, B. (2019) · International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/899 · Cited by 3
“Machine learning methods can play a key role in expediting this process by semi-automating screening [Cohen et al., 2006; Wallace et al., 2010b].”
Abògúnr̀in, Ṣ., Queiros, L., Bednarski, M., Sumner, M., Baehrens, D., & Witzmann, A. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.30.462652 · Cited by 3
“where TP and FN were previously defined above, R is recall, and N is the total number of 219 samples in the test set [22].”
BackgroundInfluentialTemporal Extraction and Retrieval
Alonso, O., & Spaniol, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674137 · Cited by 2
Tonellotto, N. (2024) · ACM eBooks · doi:10.1145/3674127.3674130 · Cited by 3
Hilkenmeier, F., Stoltenberg, M., & Stierle, C. M. G. (2026) · Systematic Reviews · doi:10.1186/s13643-026-03228-4 · Cited by 7
“Naturally, restricting automated decisions to cases of full agreement reduces the amount of work saved over sampling (e.g., [17, 60]) compared to a major-ity-vote approach.”
Using Ensemble of Bayesian Classifying Algorithms for Medical Systematic Reviews
Aref, A., & Tran, T. T. (2014) · Lecture notes in computer science · doi:10.1007/978-3-319-06483-3_23 · Cited by 2
Tompson, L. (2019) · Journal of Experimental Criminology · doi:10.1007/s11292-019-09359-0 · Cited by 2
Varghese, A., Allen, K., Agyeman-Badu, G., Haire, J. A., & Madsen, R. (2021) · Environment Systems & Decisions · doi:10.1007/s10669-021-09830-2 · Cited by 2
Digitization of Prior Authorization in Healthcare Management Using Machine Learning
Ginjupalli, S., Radhesyam, V., Manne, S., Sahithi, G., & Keerthana, S. S. (2022) · Current Signal Transduction Therapy · doi:10.2174/1574362417666220412132348 · Cited by 2
Active Learning for Biomedical Article Classification with Bag of Words and FastText Embeddings
Cichosz, P. (2024) · Applied Sciences · doi:10.3390/app14177945 · Cited by 3
Angrish, M. M., Thayer, K. A., Schulz, B., Nowak, A. J., Persad, A. S., Phillips, A. L., Rice, G., Shannon, T., Wilkins, A. A., Christensen, K. L. Y., et al. (2024) · Evidence-Based Toxicology · doi:10.1080/2833373x.2024.2421192 · Cited by 2
Verberne, S. (2024) · ACM eBooks · doi:10.1145/3674127.3674141 · Cited by 5
Ferro, N., & Maistro, M. (2024) · ACM eBooks · doi:10.1145/3674127.3674132 · Cited by 3
Privacy in Information Retrieval
Soboroff, I. M. (2024) · ACM eBooks · doi:10.1145/3674127.3674139 · Cited by 2
An NLP Based Approach to Automate and Enhance the Systematic Review Within PRISMA Format
Thankachan, J. A., & Srinivasan, B. (2024) · IFIP advances in information and communication technology · doi:10.1007/978-3-031-73617-9_16 · Cited by 2
Boolean and Ranked Information Retrieval for Biomedical Systematic Reviewing
Pohl, S. (2012) · Minerva Access (University of Melbourne) · Cited by 2
Implementing High Performance Retrieval Process by Max-Score Ranking
Vignesh, U. (2013) · IOSR Journal of Computer Engineering · doi:10.9790/0661-0852833 · Cited by 2
Crowdsourcing a Comprehensive Clinical Trial Repository
van Valkenhoef, G., & Kuiper, J. (2014) · University of Groningen research database (University of Groningen / Centre for Information Technology) · Cited by 2
Matching Possible Mitigations to Cyber Threats: A Document-Driven Decision Support Systems Approach
McNeil, M., Noteboom, C., Liu, J., El-Gayar, O., & Llansó, T. (2020) · Hawaii International Conference on System Sciences (HICSS) · doi:10.24251/hicss.2020.780 · Cited by 2
A Roadmap for Composing Automatic Literature Reviews: A Text Mining Approach
Júnior, E. M. D. S., & Dutra, M. L. (2021) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-030-77417-2_17 · Cited by 2
Hou, J., Wang, X., Dubois, J. B., Rice, R. B., Haddock, A., & Wang, Y. (2022) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3511808.3557600 · Cited by 2
“The metric is commonly used in systematic review research [7, 14].”
MethodGarritty, C. (2009) · TSpace (University of Toronto) · Cited by 2
Testing Reading Tactics for Automated Reading Assistance: Is it Useful to Apply Old Knowledge?Preprint
Yu, Z., & Menzies, T. (2017) · arXiv (Cornell University) · Cited by 2
Graph-Sparse LDA: A Topic Model with Structured SparsityPreprint
Doshi‐Velez, F., Wallace, B., & Adams, R. P. (2014) · arXiv (Cornell University) · doi:10.48550/arxiv.1410.4510 · Cited by 2
“We consider a dataset of 1218 documents annotated with 5347 unique MeSH terms (23 average terms per document) that were screened for a systematic review of the effects of calcium-channel blocker (CCB) drugs (Cohen et al. 2006).”
Understanding in vivo modelling of depression
Bannach‐Brown, A. (2019) · ERA · Cited by 1
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2023) · Research Square · doi:10.21203/rs.3.rs-3425312/v1 · Cited by 2
Romanov, S., Siqueira, A. S., Bruin, J. D., Teijema, J. J., Hofstee, L., & van de Schoot, R. (2023) · doi:10.31234/osf.io/mh4vt · Cited by 2
Romanov, S., Siqueira, A. S., Bruin, J. D., Teijema, J. J., Hofstee, L., & van de Schoot, R. (2023) · doi:10.31234/osf.io/9h5ju · Cited by 2
den Boer, R., Hofstee, L., Leenaars, C. H. C., Bagheri, A., Teijema, J. J., & van de Schoot, R. (2024) · PsyArXiv (OSF Preprints) · doi:10.31234/osf.io/zht9d · Cited by 1
LLM-based Multi-Agent Collaboration for Abstract Screening towards Automated Systematic ReviewsPreprint
Akinseloyin, O., Jiang, X., & Palade, V. (2025) · medRxiv · doi:10.1101/2025.08.11.25333429 · Cited by 2
“WSS@ R % measures the proportion of the collection that can be skipped when R % of the relevant documents are retrieved [13].”
Metadata
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
