Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Open accessAbstract
OBJECTIVE: To determine whether automated classification of document citations can be useful in reducing the time spent by experts reviewing journal articles for inclusion in updating systematic reviews of drug class efficacy for treatment of disease. DESIGN: A test collection was built using the annotated reference files from 15 systematic drug class reviews. A voting perceptron-based automated citation classification system was constructed to classify each article as containing high-quality, drug class-specific evidence or not. Cross-validation experiments were performed to evaluate performance. MEASUREMENTS: Precision, recall, and F-measure were evaluated at a range of sample weightings. Work saved over sampling at 95% recall was used as the measure of value to the review process. RESULTS: A reduction in the number of articles needing manual review was found for 11 of the 15 drug review topics studied. For three of the topics, the reduction was 50% or greater. CONCLUSION: Automated document citation classification could be a useful tool in maintaining systematic reviews of the efficacy of drug therapy. Further work is needed to refine the classification system and determine the best manner to integrate the system into the production of systematic reviews.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes (>2000 publications) where approximately less than 5% of publications are relevant.
“Previous benchmark studies showed that substantial workload reductions with active learning mostly occur in large sample sizes ( > 2000 publications) where approximately less than 5% of publications are relevant 13,20 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
The Discussion reports substantial workload savings for several reviews, but explicitly says performance had essentially no correlation with sample size or the fraction of positive samples. It therefore contradicts the claim that savings mostly occurred in collections of more than 2,000 publications with less than 5% relevant. The paper describes automated classification, not active learning. (LLM, conf=0.88)
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredMachine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning.
“Machine learning offers approaches to overcome the manual and time-consuming screening of large numbers of studies by prioritizing relevant studies via active learning 15 .”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
The Abstract says machine-learning classification could reduce manual review and help reviewers prioritize articles most likely to contain high-quality evidence. The Methods describe training a classifier on existing test collections, but our copy does not identify the approach as active learning; that specific mechanism is unsupported. (LLM, conf=0.88)
Experimental — accuracy not yet measuredIn the test collection of 15 systematic reviews, the ratio of included articles ranged from 0.5% to 21.7%, with the largest review containing 3465 studies and the smallest 310.
“The ratio of included articles ranged from 0.5 to 21.7%, with the largest review containing 3465 studies and the smallest 310.”
From Rayyan-a web and mobile app for systematic reviews. (2016) · cites this paper as [16]
Table 3 (rendered under Conclusions) lists 15 reviews and test-collection sizes ranging from 310 to 3,465 citations. It also lists a minimum included-article percentage of 0.5%, but the maximum is 27.0% for OralHypoglycemics, not 21.7%; the latter is the figure for Estrogens. The size figures count citations in the test collections, not necessarily studies. (LLM, conf=0.92)
Experimental — accuracy not yet measuredAll eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1.6% to 27% and dataset sizes from 310 to 2,544 records.
“Benchmark scope: All eight datasets are drug class reviews from the Cohen (2006) collection, covering inclusion rates from 1 . 6% to 27% and dataset sizes from 310 to 2 , 544 records.”
From Auxiliary uncertainty signals for LLM-assisted systematic review screening: a benchmark across eight Cohen drug-class reviews (2026) · cites this paper as Cohen (2006)
Table 3 (listed under Conclusions; page unavailable) identifies drug reviews with test-collection sizes of 310 and 2,544 and inclusion percentages of 1.6% and 27%. But Building the Text Collections describes 15 source datasets, not the citing paper’s eight-dataset selection; our copy does not establish which eight were selected or that those are their ranges. (LLM, conf=0.85)
Experimental — accuracy not yet measuredThe work savings from GPT-4 are higher than older results for ML and NLP classifiers and on par with more recent ML results with humans in the loop.
“The work savings are thus higher than the older results for ML and NLP classifiers [21] and on par with more recent ML results with humans in the loop [5].”
From Testing the utility of GPT for title and abstract screening in environmental systematic evidence synthesis (2025) · cites this paper as [21]
The Results (page not provided) report work savings for this older machine-learning citation classifier, including an average WSS@95% of 23.43% across 15 review topics. The Discussion says NLP-derived features were proposed for future work, not used in this classifier, so the source does not support the NLP portion. The comparison with GPT-4 is the citing authors’ own comparison, not a result attributed to this 2006 paper. (LLM, conf=0.84)
Experimental — accuracy not yet measured
The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than drug effectiveness reviews.
“The prioritisation succeeded in suggesting relevant references across a variety of reviews from the public health domain, which has less clearly defined and more complex inclusion criteria than, for example, drug effectiveness reviews.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The Discussion (page unavailable) reports substantial reviewer-effort savings for 11 of 15 reviews, supporting success across multiple reviews. But these were drug reviews, not public-health reviews, and the cited paper does not establish that public-health inclusion criteria are less clearly defined or more complex. (LLM, conf=0.89)
Experimental — accuracy not yet measuredThe fourth dataset is a static subset of 2,544 publications from the TREC 2004 Genomics Track document corpus, from all MEDLINE records 1994-2003, with 41 publications included in the review (1.6%).
“The collection is a subset of 2,544 publications from the TREC 2004 Genomics Track document corpus 48 .This is a static subset from all MEDLINE records from 1994 through 2003, which allows for replicability of results.Forty-one publications were included in the review (1.6%).”
From An open source machine learning framework for efficient and transparent systematic reviews (2021) · cites this paper as [15]
In a previous study using the ACE dataset without active learning, the WSS@95 value was 56.61%.
“In a previous study, the ACE dataset was used to simulate a model that did not use active learning, finding a WSS@95 value of 56.61% [31]”
There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature for systematic reviews.
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
From Toward good in vitro reporting standards (2019) · cites this paper as Cohen et al., 2006
With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.
“With prioritisation, it may be possible to perform a partial screening and still identify the vast majority, for example, 95%, of the relevant references.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
Machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.
“Furthermore, machine learning algorithms that learn from a human screener's decisions can be harnessed to prioritise the remaining unseen references by their predicted relevancy.9”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies and some prospective studies that have explored different machine learning approaches.
“The potential benefit of incorporating machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections9, 13, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31) and some prospective studies32, 33 that have explored different machine learning approaches.”
From Prioritising references for systematic reviews with RobotAnalyst: A user study (2018) · cites this paper as [9]
This pipeline was evaluated using the WSS metric, which quantifies the reduction in screening effort compared to random sampling while achieving a target recall level.
“This pipeline was evaluated using the WSS metric, which quanti fi es the reduction in screening effort compared to random sampling while achieving a target recall level 13 .”
From AIM review tool: artificial intelligence for smarter systematic review screening (2026) · cites this paper as [13]
A true recall of 95% is a common recall number used when measuring reduction in workload.
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
A true recall of 95% approximates the level of human error in screening.
“This number (95%) was selected as it is a common recall number used when measuring the reduction in workload; it also approximates the level of human error in screening [3, 16, 26].”
From An evaluation of DistillerSR’s machine learning-based prioritization tool for title/abstract screening – impact on reviewer-relevant outcomes (2020) · cites this paper as [26]
Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).
“Cohen, Hersh, Peterson and Yen [33] were the first to automate a part of the systematic review, specifically the Selection of Primary Studies (SLR6).”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
The article of Cohen et al. [33] was the oldest included, published in 2006.
“To add, it was also the oldest article included, being published in 2006.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
The article of Cohen et al. [33] was the most cited, with 219 citations.
“With 219 citations, the article of Cohen et al. [33] was the article most cited.”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Work Saved over Sampling (WSS) is the primary metric in this field, used 9 times, founded by [33].
“As it is used 9 times, the primary metric in this field is Work Saved over Sampling (WSS), founded by [33].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision.
“A significant impact on both the total cost of a review and the time required to produce a review can be achieved with even small increases in precision [33,46,78].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach.
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [33]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · see citing works
The perceptron: A probabilistic model for information storage and organization in the brain.
Rosenblatt, F. (1958) · Psychological Review · doi:10.1037/h0042519 · Cited by 12,280 · see citing works
An algorithm for suffix stripping
Porter, M. F. (1980) · Program · doi:10.1108/eb046814 · Cited by 8,096 · see citing works
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · see citing works
Clinical Epidemiology: A Basic Science for Clinical Medicine.
(1991) · Annals of Internal Medicine · doi:10.7326/0003-4819-114-9-825_9 · Cited by 3,497 · see citing works
The American Heritage Word Frequency Book
Warren, R. E., Carroll, J. B., Davies, P. S., & Richman, B. (1973) · The American Journal of Psychology · doi:10.2307/1421864 · Cited by 1,379 · see citing works
Large margin classification using the perceptron algorithm
Freund, Y., & Schapire, R. E. (1998) · Conference on Learning Theory · doi:10.1145/279943.279985 · Cited by 1,091 · see citing works
Postmenopausal Hormone Replacement Therapy: Scientific Review
Nelson, H. D., Humphrey, L. L., Nygren, P., Teutsch, S. M., & Allan, J. D. (2002) · JAMA · doi:10.1001/jama.288.7.872 · Cited by 990 · see citing works
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N. L., McKibbon, K. A., Walker, C. J., & Sinclair, J. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 762 · see citing works
Large Margin Classification Using the Perceptron Algorithm
Freund, Y., & Schapire, R. E. (1999) · Machine Learning · doi:10.1023/a:1007662407062 · Cited by 600 · see citing works
Chou, R., Peterson, K., & Helfand, M. (2004) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2004.05.002 · Cited by 375 · see citing works
Self-citationA simple, fast, and effective rule learner
Cohen, W. W., & Singer, Y. (1999) · Cited by 341 · see citing works
Haynes, R. B. (2002) · BMC Health Services Research · doi:10.1186/1472-6963-2-3 · Cited by 296 · see citing works
TREC 2004 Genomics Track Overview
Hersh, W., Bhuptiraju, R. T., Ross, L., Johnson, P., Cohen, A., & Kraemer, D. F. (2003) · Cited by 259 · see citing works
Self-citationA categorization and analysis of the criticisms of Evidence-Based Medicine
Cohen, A. M., Stavri, P. Z., & Hersh, W. (2003) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2003.11.002 · Cited by 256 · see citing works
Self-citationChou, R., Clark, E., & Helfand, M. (2003) · Journal of Pain and Symptom Management · doi:10.1016/j.jpainsymman.2003.03.003 · Cited by 229 · see citing works
Text Categorization Models for High-Quality Article Retrieval in Internal Medicine
Aphinyanaphongs, Y. (2004) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1641 · Cited by 177 · see citing works
Systematic reviews : synthesis of best evidence for health care decisions
Mulrow, C. D., & Cook, D. J. (1998) · American College of Physicians eBooks · Cited by 171 · see citing works
Evaluation of BioCreAtIvE assessment of task 2
Blaschke, C., Andrés‐León, E., Krallinger, M., & Valencia, A. (2005) · BMC Bioinformatics · doi:10.1186/1471-2105-6-s1-s16 · Cited by 139 · see citing works
Developing optimal search strategies for detecting sound clinical prediction studies in MEDLINE.
Wong, S. S.-L., Wilczynski, N. L., Haynes, R. B., & Ramkissoonsingh, R. (2003) · PubMed · Cited by 102 · see citing works
Developing optimal search strategies for detecting clinically sound causation studies in MEDLINE.
Wilczynski, N. L., & Haynes, R. B. (2003) · PubMed · Cited by 89 · see citing works
“A world of knowledge at your fingertips”
Hersh, W. (1999) · Academic Medicine · doi:10.1097/00001888-199903000-00012 · Cited by 52 · see citing works
Self-citationCohen, A. M., Bhupatiraju, R. T., & Hersh, W. (2004) · Cited by 38 · see citing works
Self-citationAssisting medical annotation in Swiss-Prot using statistical classifiers
Dobrokhotov, P. B., Goutte, C., Veuthey, A., & Gaussier, É. (2004) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2004.04.017 · Cited by 13 · see citing works
Flagged first, then by citations
1 of the 25 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
Rayyan-a web and mobile app for systematic reviews.
Ouzzani, M., Hammady, H., Fedorowicz, Z., & Elmagarmid, A. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0384-4 · Cited by 26,092 · see citing works
“The work saved over random sampling measured at 0.95 recall (WSS@95), introduced in [16], refers to the percentage of studies that the reviewers do not have to go through because they have been screened out by the classifier at a recall of 0.95, compared to random sampling.”
MethodResultInfluentialAn open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., de Bruin, J., Schram, R., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-020-00287-7 · Cited by 1,157 · see citing works
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · see citing works
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432 · see citing works
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · see citing works
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 351 · see citing works
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287 · see citing works
Error rates of human reviewers during abstract screening in systematic reviews
Wang, Z., Nayfeh, T., Tetzlaff, J., O’Blenis, P., & Murad, M. H. (2020) · PLOS ONE · doi:10.1371/journal.pone.0227742 · Cited by 217 · see citing works
Titles versus titles and abstracts for initial screening of articles for systematic reviews
Mateen, F., Oh, Tergas, Bhayani, & Kamdar (2013) · Clinical Epidemiology · doi:10.2147/clep.s43118 · Cited by 202 · see citing works
Applications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212 · see citing works
Can ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275 · see citing works
Wallace, B. C., Noel-Storr, A., Marshall, I. J., Cohen, A. M., Smalheiser, N. R., & Thomas, J. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx053 · Cited by 186 · see citing works
Self-citationChai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190 · see citing works
SWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196 · see citing works
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185 · see citing works
The State of the Art in Enhancing Trust in Machine Learning Models with the Use of Visualizations
Chatzimparmpas, A., Martins, R. M., Jusufi, I., Kucher, K., Fabrice Rossi, A., & Kerren, A. (2020) · Computer Graphics Forum · doi:10.1111/cgf.14034 · Cited by 184 · see citing works
TREC genomics special issue overview
Hersh, W., & Voorhees, E. M. (2008) · Information Retrieval · doi:10.1007/s10791-008-9076-6 · Cited by 180 · see citing works
Self-citationShemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170 · see citing works
Kirrane, E. F., Luben, T. J., Benson, A. F., Owens, E. O., Sacks, J. D., Dutton, S. J., Madden, M., & Nichols, J. L. (2019) · Environment International · doi:10.1016/j.envint.2019.02.027 · Cited by 167 · see citing works
Artificial intelligence to automate the systematic review of scientific literature
de la Torre-López, J., Ramírez, A., & Romero, J. R. (2023) · Computing · doi:10.1007/s00607-023-01181-x · Cited by 170 · see citing works
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157 · see citing works
“There are many examples of research from this category, including our own recent work (Cohen, 2006; Cohen et al., 2006; Martinez et al., 2008; Cohen et al., 2010; Matwin et al., 2010; Wallace et al., 2010a, 2010b; Cohen, 2011; Wallace and Small, 2011; Kim and Choi, 2012; Jonnalagadda and Petitti, 2013; Miwa et al., 20…”
MethodPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148 · see citing works
Sampson, M., Cumber, J., Li, C., Pound, C., Fuller, A., & Harrison, D. M. (2013) · PeerJ · doi:10.7717/peerj.147 · Cited by 150 · see citing works
Hamel, C., Kelly, S. E., Thavorn, K., Rice, D. B., Wells, G. A., & Hutton, B. (2020) · BMC Medical Research Methodology · doi:10.1186/s12874-020-01129-1 · Cited by 140 · see citing works
Lucini, F. R., Fogliatto, F. S., da Silveira, G. J. C., Neyeloff, J. L., Anzanello, M. J., Kuchenbecker, R. S., & Schaan, B. D. (2017) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2017.01.001 · Cited by 139 · see citing works
Sampson, M., Zhang, L., Morrison, A., Barrowman, N. J., Clifford, T., Platt, R. W., Klassen, T. P., & Moher, D. (2006) · BMC Medical Research Methodology · doi:10.1186/1471-2288-6-33 · Cited by 133 · see citing works
DeLuca, N. M., Angrish, M., Wilkins, A., Thayer, K., & Cohen Hubal, E. A. (2021) · Environment International · doi:10.1016/j.envint.2020.106308 · Cited by 122 · see citing works
Statistical stopping criteria for automated screening in systematic reviews
Callaghan, M. W., & Müller-Hansen, F. (2020) · Systematic Reviews · doi:10.1186/s13643-020-01521-4 · Cited by 121 · see citing works
An open source machine learning framework for efficient and transparent systematic reviews
van de Schoot, R., Bruin, J. D., Schram, R. D., Zahedi, P., de Boer, J., Weijdema, F., Kramer, B. M. R., Huijts, M., Hoogerwerf, M., Ferdinands, G., et al. (2021) · Utrecht University Repository (Utrecht University) · Cited by 106 · see citing works
Topic detection using paragraph vectors to support active learning in systematic reviews
Hashimoto, K., Kontonatsios, G., Miwa, M., & Ananiadou, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.06.001 · Cited by 100 · see citing works
Khabsa, M., Elmagarmid, A., Ilyas, I., Hammady, H., & Ouzzani, M. (2016) · Machine Learning · doi:10.1007/s10994-015-5535-7 · Cited by 98 · see citing works
Systematic Evidence Map for Over One Hundred and Fifty Per- and Polyfluoroalkyl Substances (PFAS)
Carlson, L. M., Angrish, M. M., Shirke, A. V., Radke, E. G., Schulz, B., Kraft, A. D., Judson, R., Patlewicz, G. Y., Blain, R. B., Lin, C. J., et al. (2022) · Environmental Health Perspectives · doi:10.1289/ehp10343 · Cited by 98 · see citing works
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96 · see citing works
“Of these six datasets, ACE and Software have been used for model simulations in previous studies on ML-aided title and abstract screening [11, 31].”
BackgroundMethodInfluentialCross-Topic Learning for Work Prioritization in Systematic Review Creation and Update
Cohen, A. M., Ambert, K., & McDonagh, M. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m3162 · Cited by 88 · see citing works
Self-citationCohen, A. M., Smalheiser, N. R., McDonagh, M. S., Yu, C., Adams, C. E., Davis, J. M., & Yu, P. S. (2015) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocu025 · Cited by 78 · see citing works
Self-citationOfori-Boateng, R., Aceves‐Martins, M., Wiratunga, N., & Moreno‐García, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 108 · see citing works
Ge, L., Agrawal, R., Singer, M., Kannapiran, P., De Castro Molina, J. A., Teow, K. L., Yap, C. W., & Abisheganaden, J. A. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02682-2 · Cited by 92 · see citing works
Using information mining of the medical literature to improve drug safety
Shetty, K. D., & Dalal, S. R. (2011) · Journal of the American Medical Informatics Association · doi:10.1136/amiajnl-2011-000096 · Cited by 85 · see citing works
Finding better active learners for faster literature reviews
Yu, Z., Kraft, N. A., & Menzies, T. (2018) · Empirical Software Engineering · doi:10.1007/s10664-017-9587-0 · Cited by 83 · see citing works
Supporting systematic reviews using LDA-based document representations
Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0117-0 · Cited by 86 · see citing works
Scalability of Continuous Active Learning for Reliable High-Recall Text Classification
Cormack, G. V., & Grossman, M. R. (2016) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/2983323.2983776 · Cited by 81 · see citing works
FAST2: An intelligent assistant for finding relevant papers
Yu, Z., & Menzies, T. (2019) · Expert Systems with Applications · doi:10.1016/j.eswa.2018.11.021 · Cited by 80 · see citing works
Precision of healthcare systematic review searches in a cross‐sectional sample
Sampson, M., Tetzlaff, J., & Urquhart, C. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.42 · Cited by 80 · see citing works
(2024) · ACM eBooks · doi:10.1145/3674127 · Cited by 82 · see citing works
Boolean versus ranked querying for biomedical systematic reviews
Karimi, S., Pohl, S., Scholer, F., Cavedon, L., & Zobel, J. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-58 · Cited by 79 · see citing works
Screening articles for systematic reviews with ChatGPT
Syriani, E., David, I., & Kumar, G. (2024) · Journal of Computer Languages · doi:10.1016/j.cola.2024.101287 · Cited by 82 · see citing works
Text-Mining Techniques and Tools for Systematic Literature Reviews: A Systematic Literature Review
Feng, L., Chiam, Y. K., & Lo, S. K. (2017) · Asia-Pacific Software Engineering Conference (APSEC) · doi:10.1109/apsec.2017.10 · Cited by 75 · see citing works
A new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · see citing works
Combining Crowd and Expert Labels Using Decision Theoretic Active Learning
Nguyen, A., Wallace, B., & Lease, M. (2015) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v3i1.13225 · Cited by 74 · see citing works
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · see citing works
Iqbal, S., Hassan, S., Aljohani, N. R., Alelyani, S., Nawaz, R., & Bornmann, L. (2021) · Scientometrics · doi:10.1007/s11192-021-04055-1 · Cited by 77 · see citing works
Neto, F. G. D. O., Torkar, R., FELDT, R. H., Gren, L., Furia, C. A., & Huang, Z. (2019) · Journal of Systems and Software · doi:10.1016/j.jss.2019.07.002 · Cited by 71 · see citing works
Improving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62 · see citing works
Machine learning to assist risk-of-bias assessments in systematic reviews
Millard, L. A., Flach, P. A., & Higgins, J. P. (2016) · International Journal of Epidemiology · doi:10.1093/ije/dyv306 · Cited by 65 · see citing works
Scells, H., Zuccon, G., Koopman, B., Deacon, A. J., Azzopardi, L. A., & Geva, S. (2017) · ACM International Conference on Information and Knowledge Management (CIKM) · doi:10.1145/3132847.3133080 · Cited by 68 · see citing works
Sim, I., Tu, S. W., Carini, S., Lehmann, H. P., Pollock, B. H., Mor Peleg, B., & Wittkowski, K. M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.11.002 · Cited by 68 · see citing works
Development of Prompt Templates for Large Language Model–Driven Screening in Systematic Reviews
Cao, C., Sang, J., Arora, R., Chen, D., Kloosterman, R., Cecere, M., Gorla, J., Saleh, R., Drennan, I., Teja, B., et al. (2025) · Annals of Internal Medicine · doi:10.7326/annals-24-02189 · Cited by 67 · see citing works
Recent Progress in Automatically Extracting Information from the Pharmacogenomic Literature
Garten, Y., Coulet, A., & Altman, R. B. (2010) · Pharmacogenomics · doi:10.2217/pgs.10.136 · Cited by 66 · see citing works
The fourteenth text retrieval conference TREC 2005
Voorhees, E. M., & Buckland, L. P. (2006) · doi:10.6028/nist.sp.500-266 · Cited by 66 · see citing works
Keshava, C., Nicolai, S., Vulimiri, S. V., Cruz, F. A., Ghoreishi, N., Knüppel, S., Lenzner, A., Tarnow, P., Vanselow, J. T., Schulz, B., et al. (2023) · Environment International · doi:10.1016/j.envint.2023.107952 · Cited by 65 · see citing works
A new iterative method to reduce workload in systematic review process
Jonnalagadda, S., & Petitti, D. (2013) · International Journal of Computational Biology and Drug Design · doi:10.1504/ijcbdd.2013.052198 · Cited by 61 · see citing works
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 65 · see citing works
Toward good in vitro reporting standards
Hartung, T. (2019) · ALTEX · doi:10.14573/altex.1812191 · Cited by 63 · see citing works
“There is growing interest in using machine learning approaches to priority rank studies and reduce human burden in screening literature (Cohen et al., 2006) when conducting systematic reviews.”
BackgroundFeng, Y., Liang, S., Zhang, Y., Chen, S., Wang, Q., Huang, T., Sun, F., Liu, X., Zhu, H., & Pan, H. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac066 · Cited by 66 · see citing works
Optimizing feature representation for automated systematic review work prioritization.
Cohen, A. M. (2008) · PubMed Central · Cited by 62 · see citing works
Self-citationShirke, A. V., Radke, E. G., Lin, C. J., Blain, R. B., Vetter, N., Lemeris, C., Hartman, P. A., Hubbard, H. F., Angrish, M. M., Arzuaga, X., et al. (2024) · Environmental Health Perspectives · doi:10.1289/ehp13423 · Cited by 61 · see citing works
Patient healthcare trajectory. An essential monitoring tool: a systematic review
Pinaire, J., Azé, J., Bringay, S., & Landais, P. (2017) · Health Information Science and Systems · doi:10.1007/s13755-017-0020-2 · Cited by 63 · see citing works
Towards Automating the Initial Screening Phase of a Systematic Review
Bekhuis, T., & Demner‐Fushman, D. (2010) · Studies in health technology and informatics · doi:10.3233/978-1-60750-588-4-146 · Cited by 61 · see citing works
Improving class probability estimates for imbalanced data
Wallace, B. C., & Dahabreh, I. (2014) · Knowledge and Information Systems · doi:10.1007/s10115-013-0670-6 · Cited by 74 · see citing works
A Test Collection for Evaluating Retrieval of Studies for Inclusion in Systematic Reviews
Scells, H., Zuccon, G., Koopman, B., Deacon, A. J., Azzopardi, L. A., & Geva, S. (2017) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3077136.3080707 · Cited by 59 · see citing works
Automating Risk of Bias Assessment for Clinical Trials
Marshall, I. J., Kuiper, J., & Wallace, B. C. (2015) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2015.2431314 · Cited by 59 · see citing works
Bao, Y., Deng, Z., Wang, Y., Kim, H., Armengol, V. D., Francisco Acevedo, C., Ouardaoui, N., Wang, C., Parmigiani, G., Barzilay, R., et al. (2019) · JCO Clinical Cancer Informatics · doi:10.1200/cci.19.00042 · Cited by 59 · see citing works
Generating Better Queries for Systematic Reviews
Scells, H., & Zuccon, G. (2018) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3209978.3210020 · Cited by 55 · see citing works
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54 · see citing works
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51 · see citing works
Thomas, J. (2013) · OA Evidence-Based Medicine · doi:10.13172/2053-2636-1-2-1109 · Cited by 44 · see citing works
Modernizing the systematic review process to inform comparative effectiveness: tools and methods
Wallace, B., Dahabreh, I. J., Schmid, C. H., Lau, J., & Trikalinos, T. A. (2013) · Journal of Comparative Effectiveness Research · doi:10.2217/cer.13.17 · Cited by 51 · see citing works
Crowdsourcing the Citation Screening Process for Systematic Reviews: Validation Study
Nama, N., Sampson, M., Barrowman, N. J., Sandarage, R. V., Menon, K., Macartney, G. M., Murto, K., Vaccani, J., Katz, S. L., Zemek, R. L., et al. (2019) · Journal of Medical Internet Research · doi:10.2196/12953 · Cited by 50 · see citing works
Automatic Boolean Query Formulation for Systematic Review Literature Search
Scells, H., Zuccon, G., Koopman, B., & Clark, J. (2020) · The Web Conference · doi:10.1145/3366423.3380185 · Cited by 56 · see citing works
Hanegraaf, P., Wondimu, A., Mosselman, J. J., de Jong, R., Abògúnr̀in, Ṣ., Queiros, L., Lane, M., Postma, M. J., Boersma, C., & van der Schans, J. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-076912 · Cited by 57 · see citing works
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 53 · see citing works
Ji, X., Shen, H., Ritter, A., Machiraju, R., & Yen, P. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2903946 · Cited by 48 · see citing works
Self-citationRefining Boolean queries to identify relevant studies for systematic review updates
Alharbi, A., & Stevenson, M. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa148 · Cited by 47 · see citing works
Cohen, A. M., Adams, C. E., Davis, J. M., Yu, C. T., Yu, P. S., Meng, W., Duggan, L., McDonagh, M. S., & Smalheiser, N. R. (2010) · International Health Informatics Symposium · doi:10.1145/1882992.1883046 · Cited by 47 · see citing works
Self-citationJi, X., Ritter, A., & Yen, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.03.007 · Cited by 48 · see citing works
Self-citationA Systematic Review of Virtual Reality Therapeutics for Acute Pain Management
Dreesmann, N. J., Su, H., & Thompson, H. J. (2022) · Pain Management Nursing · doi:10.1016/j.pmn.2022.05.004 · Cited by 47 · see citing works
Exploiting the systematic review protocol for classification of medical abstracts
Frunza, O., Inkpen, D. Z., Matwin, S., Klement, W., & O’Blenis, P. (2010) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2010.10.005 · Cited by 45 · see citing works
Odintsova, V. V., Roetman, P. J., Ip, H. F., Pool, R., van der Laan, C. M., Tona, K., Vermeiren, R. R. J. M., & Boomsma, D. I. (2019) · Psychiatric Genetics · doi:10.1097/ypg.0000000000000239 · Cited by 44 · see citing works
Graph-Sparse LDA: A Topic Model with Structured Sparsity
Doshi‐Velez, F., Wallace, B., & Adams, R. P. (2015) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v29i1.9603 · Cited by 45 · see citing works
Zhang, Y., Liang, S., Feng, Y., Wang, Q., Sun, F., Chen, S., Yang, Y., He, X., Zhu, H., & Pan, H. (2022) · Systematic Reviews · doi:10.1186/s13643-021-01881-5 · Cited by 49 · see citing works
Carlson, L. M., Christensen, K. L. Y., Sagiv, S. K., Rajan, P. K., Klocke, C. R., Lein, P. J., Coffman, E., Shaffer, R. M., Yost, E. E., Arzuaga, X., et al. (2022) · Environmental Research · doi:10.1016/j.envres.2022.115148 · Cited by 44 · see citing works
Cohen, A. M. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2434 · Cited by 44 · see citing works
Self-citationNama, N., Hennawy, M., Barrowman, N., O’Hearn, K., Sampson, M., & McNally, J. D. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01632-6 · Cited by 50 · see citing works
The current state of systematic reviews in library and information studies
Xu, J., Kang, Q., & Song, Z. (2015) · Library & Information Science Research · doi:10.1016/j.lisr.2015.11.003 · Cited by 42 · see citing works
A semi-supervised approach using label propagation to support citation screening
Kontonatsios, G., Brockmeier, A. J., Przybyła, P., McNaught, J., Mu, T., Goulermas, J. Y., & Ananiadou, S. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.06.018 · Cited by 42 · see citing works
Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42 · see citing works
Linked data approach for selection process automation in systematic reviews
Tomassetti, F., Rizzo, G. F., Vetrò, A., Ardito, L., Torchiano, M., & Morisio, M. (2011) · International Conference on Evaluation and Assessment in Software Engineering (EASE) · doi:10.1049/ic.2011.0004 · Cited by 41 · see citing works
A comparison of automatic Boolean query formulation for systematic reviews
Scells, H., Zuccon, G., & Koopman, B. (2020) · Information Retrieval · doi:10.1007/s10791-020-09381-1 · Cited by 42 · see citing works
A roadmap toward the automatic composition of systematic literature reviews
Júnior, E. M. D. S., & Dutra, M. L. (2021) · Iberoamerican Journal of Science Measurement and Communication · doi:10.47909/ijsmc.52 · Cited by 40 · see citing works
An SVM-based high-quality article classifier for systematic reviews
Kim, S., & Choi, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.10.005 · Cited by 39 · see citing works
- Authors
- Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y.
- Affiliations
- Oregon Health & Science University (US)
- Corresponding author
- Cohen, A. M.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2006-03-01
- Volume
- 13
- Issue
- 2
- Pages
- 206-219
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC1447545
- OpenAlex
- W168362576
- Semantic Scholar
- 9a1a1d6f0ab114d14fc69a91f1981cf8e0848c84
- Cited by
- 411
- References
- 25
- Citations by year
- 2026: 29 · 2025: 36 · 2024: 60 · 2023: 42 · 2022: 28 · 2021: 33 · 2020: 28 · 2019: 22 · 2018: 14 · 2017: 17 · 2016: 11 · 2015: 22 · 2014: 12 · 2013: 17 · 2012: 7
- FWCI
- 0.56
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- National Science Foundation (0325160)
- National Institutes of Health
- U.S. National Library of Medicine (2 T15 LM 07088-11, T15LM007088)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Keywords
- systematic review updating, drug class review, drug therapy efficacy
- MeSH terms
- Systematic Reviews as Topic, Artificial Intelligence, Pharmaceutical Preparations, Publications, Meta-Analysis as Topic, Bibliometrics, Information Storage and Retrieval, Workload, Evidence-Based Medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
