The DOI matches its Crossref record; 24 of the 25 references it cites were found in a registry or an index.
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
Experimental — accuracy not yet measured
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The cited source says what the claim attributes to it.
In the citing paper
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
The cited source says what the claim attributes to it.
In the citing paper
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
The cited source says what the claim attributes to it.
In the citing paper
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The cited source says what the claim attributes to it.
In the citing paper
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
In the citing paper
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 622
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- 2005, BMC bioinformatics [electronic resource], 6, S16
- 1980, PROGRAM, 14, 127
Cited by
Sorted by citations
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092
“The work saved over random sampling measured at 0.95 recall (WSS@95), introduced in [16], refers to the percentage of studies that the reviewers do not have to go through because they have been screened out by the classifier at a recall of 0.95, compared to random sampling.”
MethodResultInfluentialAn open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
“Cohen et al. also reported good results for a weighted model, in which they modified their voting perceptron classifier to incorporate a false negative learning rate (FNLR) [36].”
BackgroundMethodInfluentialAutomation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
BackgroundMethodInfluentialSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
“demonstrated that machine learning techniques can reduce the labor required to update systematic reviews [16].”
BackgroundO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
“Cohen et al. [54] constructed a voting perceptron-based automated citation classification system to classify each article as to whether it contains high-quality, drug-specific evidence.”
BackgroundError rates of human reviewers during abstract screening in systematic reviews
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227742 · Cited by 217
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275
Titles versus titles and abstracts for initial screening of articles for systematic reviews
Mateen, F., Oh, Tergas, Bhayani, & Kamdar (2013) · Clinical Epidemiology · doi:10.2147/clep.s43118 · Cited by 202
“strategies for performing systematic reviews (Table 1, 2).(6,7) Precision is defined here as:(6)”
Wallace, B. C., Noel-Storr, A., Marshall, I. J., Cohen, A. M., Smalheiser, N. R., & Thomas, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx053 · Cited by 186
Self-citationChai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
“Performance was assessed using work saved over random sampling (WSS [11]), which provides an indication of the time saved over manual screening in the form a percentage reduction in the number of papers needed to be screened.”
MethodSWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196
“Antihistamines [6] PubMed (PMIDs) 310 16 (5.”
BackgroundMethodInfluentialBannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
“Machine learning approaches have been evaluated in context of systematic reviews of several medical problems including drug class efficacy assessment [7, 8, 12], genetic associations [9], public health [13, 16], cost-effectiveness analyses [9], toxicology [3], treatment effectiveness [17, 18], and nutrition [17].”
BackgroundThe State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Fabrice Rossi, A., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 204
TREC genomics special issue overview
Hersh, W., & Voorhees, E. M. (2008) · Information Retrieval · doi:10.1007/s10791-008-9076-6 · Cited by 180
Self-citationShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
“This study expands an emerging corpus of empirical evidence for the use of TM to support study screening and selection in reviews (Cohen et al., 2012; Cohen et al., 2010; Wallace et al., 2010a; Cohen et al., 2006) by demonstrating its feasibility, strengths and limitations in two extremely large-scale scoping reviews…”
BackgroundKirrane, E. F., Luben, T. J., Benson, A. F., Owens, E. O., Sacks, J. D., Dutton, S. J., Madden, M., & Nichols, J. L. (2019) · Environment International · doi:10.1016/j.envint.2019.02.027 · Cited by 167
“Using this method, references were queued based on predicted relevance, until a 95% threshold was reached (Cohen et al., 2006; Howard et al., 2016).”
MethodArtificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170
“From a historical perspective, it is also interesting to note that the selection of primary studies continues to attract attention since the publication of the first paper [8].”
BackgroundMethodInfluentialHoward, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
“There are many examples of research from this category, including our own recent work (Cohen, 2006; Cohen et al., 2006; Martinez et al., 2008; Cohen et al., 2010; Matwin et al., 2010; Wallace et al., 2010a, 2010b; Cohen, 2011; Wallace and Small, 2011; Kim and Choi, 2012; Jonnalagadda and Petitti, 2013; Miwa et al., 20…”
MethodPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Sampson, M., Cumber, J., Li, C., Pound, C., Fuller, A., & Harrison, D. M. (2013) · PeerJ · doi:10.7717/peerj.147 · Cited by 150
“(Cohen et al. 2006; Sampson et al. 2006) Yet, we were unable to identify any practical guidance on stopping rules for screening in the systematic review context, nor any explicit reports of when or if screening was stopped for Internet-based searches in systematic reviews.”
BackgroundHamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140
Lucini, F. R., Fogliatto, F. S., da Silveira, G. J. C., Neyeloff, J. L., Anzanello, M. J., Kuchenbecker, R. S., & Schaan, B. D. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.01.001 · Cited by 139
Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T., Platt, R. W., Klassen, T. P., & Moher, D. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-33 · Cited by 133
DeLuca, N. M., Angrish, M., Wilkins, A., Thayer, K., & Cohen Hubal, E. A. (2021) · Environment International · doi:10.1016/j.envint.2020.106308 · Cited by 122
Statistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106
Topic detection using paragraph vectors to support active learning in systematic reviews
Hashimoto, K., Kontonatsios, G., Miwa, M., & Ananiadou, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.06.001 · Cited by 103
“Previous approaches [6] used the WSS@95% metric to evaluate the performance of automatic classification approaches that takes into consideration only automatic screening decisions.”
BackgroundMethodKhabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., & Ouzzani, M. (2016) · Machine Learning · doi:10.1007/s10994-015-5535-7 · Cited by 98
“The dataset used in this work was created by Cohen et al. (2006).”
BackgroundMethodInfluentialSystematic Evidence Map for Over One Hundred and Fifty Per- and Polyfluoroalkyl Substances (PFAS)
Carlson, L. M., Angrish, M. M., Shirke, A. V., Radke, E. G., Schulz, B., Kraft, A. D., Judson, R., Patlewicz, G. Y., Blain, R. B., Lin, C. J., et al. (2022) · Environmental Health Perspectives · doi:10.1289/ehp10343 · Cited by 101
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96
“Of these six datasets, ACE and Software have been used for model simulations in previous studies on ML-aided title and abstract screening [11, 31].”
InfluentialCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88
“For example, in our previous work, we noticed that “CI” (an abbreviation for confidence interval) was a strong, positive predictor for some topics.(13) It is likely that this term is general enough that it applies to many SR topics.”
BackgroundMethodSelf-citationCohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78
Self-citationOfori-Boateng, R., Aceves‐Martins, M., Wiratunga, N., & Moreno‐García, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 123
Ge, L., Agrawal, R., Singer, M., Kannapiran, P., De Castro Molina, J. A., Teow, K. L., Yap, C. W., & Abisheganaden, J. A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02682-2 · Cited by 92
“Since 2005, artificial intelligence (AI) tools have gained increasing attention [1] for their ability to automate these processes, offering increased efficiency and accuracy [2].”
BackgroundUsing information mining of the medical literature to improve drug safety
Shetty, K. D., & Dalal, S. R. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000096 · Cited by 85
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83
“…mining into citation screening as well as the proposed performance metrics of Cohen has been widely acknowledged [49], most of Cohen’s work focused on supervised learning which does not utilize unlabeled data and relies on random sampling to obtain the sufficiently large training set [10,7,9,8].”
InfluentialSupporting systematic reviews using LDA-based document representations
Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0117-0 · Cited by 91
“Cohen et al. [33] constructed a voting perceptron-based automated citation classification system which is able to reduce the number of articles that needs to be reviewed by more than 50 %.”
BackgroundScalability of Continuous Active Learning for Reliable High-Recall Text Classification
Cormack, G. V., & Grossman, M. R. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983776 · Cited by 81
“The same problem has been observed within the context of systematic review in evidence-based medicine [22], but the literature has limited discussion of technological solutions to this problem [6, 40].”
BackgroundFAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80
Precision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80
“Although several groups are exploring machine learning techniques to automate screening with some success (Cohen et al., 2006; Ananiadou et al., 2009; Bekhuis and mner-Fushman, 2010; Wallace et al., 2010), human effort will be required for the foreseeable future.”
Background(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 86
Boolean versus ranked querying for biomedical systematic reviews
Karimi, S., Pohl, S., Scholer, F., Cavedon, L., & Zobel, J. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-58 · Cited by 80
“For our experiments, we used a publicly available data-set 8 of 15 systematic reviews, created by the US Agency for Healthcare Research and Quality (AHRQ) on drug-related topics, assembled by Cohen et al. [19] – we call this the Drug dataset.”
MethodScreening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82
Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review
Feng, L., Chiam, Y. K., & Lo, S. K. (2017) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.10 · Cited by 81
“[29] also applied ML to SLR, focusing on reducing the workload of reviewers during broad screening by eliminating as many non-relevant documents as possible while including no less than 95% of the relevant documents.”
BackgroundMethodInfluentialA new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100
“The classi fi er performance scores they obtained with the voting perceptron (VP) algo-rithm, 10 the algorithm that they used for the classi fi cation task, are used as a baseline for our current research.”
BackgroundMethodInfluentialCombining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 74
“Previous work has considered semi-automating this process via machine learning (Wallace et al. 2010b; Cohen et al. 2006).”
MethodScreening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70
“In general, our motivation for conducting this research is similar to that of other groups [13, 22–25], i.e., we want to facilitate production of systematic reviews.”
BackgroundResultInfluentialIqbal, S., Hassan, S., Aljohani, N. R., Alelyani, S., Nawaz, R., & Bornmann, L. (2021) · Scientometrics · doi:10.1007/s11192-021-04055-1 · Cited by 81
Neto, F. G. D. O., Torkar, R., FELDT, R. H., Gren, L., Furia, C. A., & Huang, Z. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.07.002 · Cited by 79
“[14] made the first attempts in applying text mining technologies to reduce the screening burden during reviews; more recently, a systematic review [72] analyzing text mining in systematic reviews indicated that the application of text mining technologies can reduce the workload of systematic reviews by 30–70% (while…”
BackgroundImproving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
“[14].”
InfluentialMachine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65
Scells, H., Zuccon, G., Koopman, B., Deacon, A. J., Azzopardi, L. A., & Geva, S. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133080 · Cited by 74
Sim, I., Tu, S. W., Carini, S., Lehmann, H. P., Pollock, B. H., Mor Peleg, B., & Wittkowski, K. M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.11.002 · Cited by 70
“Second, while there have been heuristic and machine-learning approaches to determining the relevance of studies to a search query [96,97] many of the approaches require training a classifier against a hand-coded gold standard, which limits their scalability.”
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 91
Recent Progress in Automatically Extracting Information from the Pharmacogenomic Literature
Garten, Y., Coulet, A., & Altman, R. B. (2010) · Pharmacogenomics · doi:10.2217/pgs.10.136 · Cited by 66
“Cohen et al. developed a voting perceptron-based citation classification system to assist production of systematic drug class evidence reviews by selecting the papers with the highest likelihood of containing high-quality evidence [13].”
BackgroundThe fourteenth text retrieval conference TREC 2005
Voorhees, E. M., & Buckland, L. P. (2006) · doi:10.6028/nist.sp.500-266 · Cited by 66
Keshava, C., Nicolai, S., Vulimiri, S. V., Cruz, F. A., Ghoreishi, N., Knüppel, S., Lenzner, A., Tarnow, P., Vanselow, J. T., Schulz, B., et al. (2023) · Environment International · doi:10.1016/j.envint.2023.107952 · Cited by 68
“…indicated that it was likely that at least 95% of the relevant studies were identified, a percent identification often used to evaluate the performance of machine learning applications and considered comparable to human error rates (Bannach-Brown et al. 2019; Cohen et al. 2006; Howard et al. 2016).”
A new iterative method to reduce workload in systematic review process
Jonnalagadda, S., & Petitti, D. (2013) · International Journal of Computational Biology and Drug Design · doi:10.1504/ijcbdd.2013.052198 · Cited by 61
“The work saved over sampling at X%, or WSS@X%, also defined by Cohen et al (A.M. Cohen et al, 2006), will be used for evaluation: Equation 4 where TN is the number of true negatives identified by the system, FN is the number of false negatives identified by the system, N is the total number of…”
MethodInfluentialdos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 70
Toward good in vitro reporting standards
Hartung, T. (2019) · ALTEX · doi:10.14573/altex.1812191 · Cited by 65
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
BackgroundFeng, Y., Liang, S., Zhang, Y., Chen, S., Wang, Q., Huang, T., Sun, F., Liu, X., Zhu, H., & Pan, H. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac066 · Cited by 72
“12 For literature screening, automated classification systems 13 or hybrid relevance rating models 14 have been evaluated in specific datasets, requiring further extension and performance improvement.”
BackgroundOptimizing feature representation for automated systematic review work prioritization.
Cohen, A. M. (2008) · PubMed Central · Cited by 62
Self-citationShirke, A. V., Radke, E. G., Lin, C. J., Blain, R. B., Vetter, N., Lemeris, C., Hartman, P. A., Hubbard, H. F., Angrish, M. M., Arzuaga, X., et al. (2024) · Environmental Health Perspectives · doi:10.1289/ehp13423 · Cited by 63
“This threshold is comparable to human error rates 37,49,50 and is used as a metric to evaluate machine-learning performance.”
Patient healthcare trajectory. An essential monitoring tool: a systematic review
Pinaire, J., Azé, J., Bringay, S., & Landais, P. (2017) · Health Information Science and Systems · doi:10.1007/s13755-017-0020-2 · Cited by 63
“Text mining allows better targeting for information retrieval and reduces the search time [11], while also enabling users to prioritise searches.”
BackgroundTowards Automating the Initial Screening Phase of a Systematic Review
Bekhuis, T., & Demner‐Fushman, D. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 61
Improving class probability estimates for imbalanced data
Wallace, B. C., & Dahabreh, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0670-6 · Cited by 74
“Specifically, the latter 22 datasets represent ‘abstract screening’ datasets [8,29].”
A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews
Scells, H., Zuccon, G., Koopman, B., Deacon, A. J., Azzopardi, L. A., & Geva, S. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080707 · Cited by 62
“WSS measures the work saved (wrt. the number of studies required to be screened) by comparing the number of not relevant studies that have not been retrieved (true negatives), those that have been retrieved, and recall.”
InfluentialAutomating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59
Bao, Y., Deng, Z., Wang, Y., Kim, H., Armengol, V. D., Francisco Acevedo, C., Ouardaoui, N., Wang, C., Parmigiani, G., Barzilay, R., et al. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00042 · Cited by 61
Generating Better Queries for Systematic Reviews
Scells, H., & Zuccon, G. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210020 · Cited by 55
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
“We conducted experiments using the simple Perceptron and SVM algorithms300 in Python’s ‘sklearn’ package [67] and the implementation of the ‘votedperceptron’ algorithm provided in Weka (with no weighting) [68], which is the algorithm that was modified in [36].”
BackgroundMethodResultInfluentialActive learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51
“WSS indicates the reduction in publications needed to be screened, at a given level of recall [32].”
InfluentialThomas, J. (2013) · OA Evidence-Based Medicine · doi:10.13172/2053-2636-1-2-1109 · Cited by 44
Modernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 52
Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study
Nama, N., Sampson, M., Barrowman, N. J., Sandarage, R. V., Menon, K., Macartney, G. M., Murto, K., Vaccani, J., Katz, S. L., Zemek, R. L., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12953 · Cited by 49
“Although no consensus study (eg, delphi, survey) has defined the minimum acceptable sensitivity, 95% has become the industry standard in the field of automated text recognition research based on original studies by Cohen [11,36].”
Automatic Boolean Query Formulation for Systematic Review Literature Search
Scells, H., Zuccon, G., Koopman, B., & Clark, J. (2020) · The Web Conference · doi:10.1145/3366423.3380185 · Cited by 56
“Note that even small increases in precision can have a significant impact on both the total cost of a review, and the time required to produce a review [5, 42].”
BackgroundMethodInfluentialHanegraaf, P., Wondimu, A., Mosselman, J. J., de Jong, R., Abògúnr̀in, Ṣ., Queiros, L., Lane, M., Postma, M. J., Boersma, C., & van der Schans, J. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-076912 · Cited by 57
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 61
“The performance of automation tools varies largely across review topics [32, 39, 54, 122, 130].”
BackgroundMethodInfluentialJi, X., Shen, H., Ritter, A., Machiraju, R., & Yen, P. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2903946 · Cited by 48
“two human experts (thus these documents are considered gold-standards or benchmark relevant documents) [29].”
BackgroundMethodInfluentialSelf-citationRefining Boolean queries to identify relevant studies for systematic review updates
Alharbi, A., & Stevenson, M. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa148 · Cited by 58
“identification of studies for inclusion in systematic reviews by applying text mining techniques, for example, [9-16].”
BackgroundCohen, A. M., Adams, C. E., Davis, J. M., Yu, C. T., Yu, P. S., Meng, W., Duggan, L., McDonagh, M. S., & Smalheiser, N. R. (2010) · International Health Informatics Symposium · doi:10.1145/1882992.1883046 · Cited by 47
Self-citationJi, X., Ritter, A., & Yen, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.03.007 · Cited by 48
Self-citationA Systematic Review of Virtual Reality Therapeutics for Acute Pain Management
Dreesmann, N. J., Su, H., & Thompson, H. J. (2022) · Pain Management Nursing · doi:10.1016/j.pmn.2022.05.004 · Cited by 47
Exploiting the systematic review protocol for classification of medical abstracts
Frunza, O., Inkpen, D. Z., Matwin, S., Klement, W., & O’Blenis, P. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.10.005 · Cited by 45
Odintsova, V. V., Roetman, P. J., Ip, H. F., Pool, R., van der Laan, C. M., Tona, K., Vermeiren, R. R. J. M., & Boomsma, D. I. (2019) · Psychiatric Genetics · doi:10.1097/ypg.0000000000000239 · Cited by 49
Bekhuis, T., Tseytlin, E., Mitchell, K. J., & Demner-Fushman, D. (2014) · PLoS ONE · doi:10.1371/journal.pone.0086277 · Cited by 45
“An interesting plan for a text-mining pipeline to support SR production is described by Cohen et al [19] with modules to reduce screening burden.”
BackgroundGraph-Sparse LDA: A Topic Model with Structured Sparsity
Doshi‐Velez, F., Wallace, B., & Adams, R. P. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9603 · Cited by 45
“We consider a dataset of 1218 documents annotated with 5347 unique MeSH terms (23 average terms per document) that were screened for a systematic review of the effects of calcium-channel blocker (CCB) drugs (Cohen et al. 2006).”
Zhang, Y., Liang, S., Feng, Y., Wang, Q., Sun, F., Chen, S., Yang, Y., He, X., Zhu, H., & Pan, H. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01881-5 · Cited by 52
“Automated literature classification systems [17] or hybrid relevance rating models [18] were tested in specific datasets, yet further extension of review datasets and performance improvement are required.”
BackgroundCarlson, L. M., Christensen, K. L. Y., Sagiv, S. K., Rajan, P. K., Klocke, C. R., Lein, P. J., Coffman, E., Shaffer, R. M., Yost, E. E., Arzuaga, X., et al. (2022) · Environmental Research · doi:10.1016/j.envres.2022.115148 · Cited by 46
“This threshold is comparable to human error rates (Bannach-Brown et al., 2018; Howard et al., 2016; Cohen et al., 2006) and is used as a metric to evaluate ML performance.”
BackgroundCohen, A. M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2434 · Cited by 44
“For example, “positive” may mean that a certain journal article is likely to be included in a systematic review of medical pain therapeutics.(1) Alternately, a “positive” label may indicate that the document should be sent to an expert biologist for annotation, who will then codify the new contributions to biomedical…”
BackgroundSelf-citationNama, N., Hennawy, M., Barrowman, N., O’Hearn, K., Sampson, M., & McNally, J. D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01632-6 · Cited by 51
“There is no clearly defined threshold for acceptable sensitivity in the literature; however, multiple studies in field of innovative methodology of SR screening have used a sensitivity of 95% as cutoff [20, 21].”
BackgroundThe current state of systematic reviews in library and information studies
Xu, J., Kang, Q., & Song, Z. (2015) · Library & Information Science Research · doi:10.1016/j.lisr.2015.11.003 · Cited by 46
A semi-supervised approach using label propagation to support citation screening
Kontonatsios, G., Brockmeier, A. J., Przybyła, P., McNaught, J., Mu, T., Goulermas, J. Y., & Ananiadou, S. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.018 · Cited by 41
“Such imbalanced datasets constitute challenging cases for machine learning methods [2,8,13,31].”
BackgroundMethodInfluentialOlorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42
“Mean recall was 95% and above in [4, 5, 17, 30, 48, 56, 74, 77, 82] while it was below 95% in [9, 49].”
BackgroundMethodInfluentialLinked data approach for selection process automation in systematic reviews
Tomassetti, F., Rizzo, G. F., Vetrò, A., Ardito, L., Torchiano, M., & Morisio, M. (2011) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2011.0004 · Cited by 41
“Cohen et al. [2006] experimented automatic classification in fifteen different systematic reviews, each one considering the validity of a particular drug.”
BackgroundMethodInfluentialA comparison of automatic Boolean query formulation for systematic reviews
Scells, H., Zuccon, G., & Koopman, B. (2020) · Information Retrieval · doi:10.1007/s10791-020-09381-1 · Cited by 42
“…screening prioritisation (Scells et al. 2020a; Kanoulas et al. 2017, 2018; Lee and Sun 2018), active learning for study inclusion classification (Cohen et al. 2006; Miwa et al. 2014), text mining (O’Mara-Eves et al. 2015; Olorisade et al. 2016; Shemilt et al. 2014), and automated second…”
BackgroundA roadmap toward the automatic composition of systematic literature reviews
Júnior, E. M. D. S., & Dutra, M. L. (2021) · Iberoamerican Journal of Science Measurement and Communication · doi:10.47909/ijsmc.52 · Cited by 45
“Khabsa et al. (2016) say that the first study to consider this possibility was by Cohen et al. (2006).”
Metadata
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
