The DOI matches its Crossref record; 6 of the 6 references it cites were found in a registry or an index.
Toward modernizing the systematic review pipeline in genetics: efficient updating via data mining
Open accessAbstract
PURPOSE: The aim of this study was to demonstrate that modern data mining tools can be used as one step in reducing the labor necessary to produce and maintain systematic reviews. METHODS: We used four continuously updated, manually curated resources that summarize MEDLINE-indexed articles in entire fields using systematic review methods (PDGene, AlzGene, and SzGene for genetic determinants of Parkinson disease, Alzheimer disease, and schizophrenia, respectively; and the Tufts Cost-Effectiveness Analysis (CEA) Registry for cost-effectiveness analyses). In each data set, we trained a classification model on citations screened up until 2009. We then evaluated the ability of the model to classify citations published in 2010 as "relevant" or "irrelevant" using human screening as the gold standard. RESULTS: Classification models did not miss any of the 104, 65, and 179 eligible citations in PDGene, AlzGene, and SzGene, respectively, and missed only 1 of 79 in the CEA Registry (100% sensitivity for the first three and 99% for the fourth). The respective specificities were 90, 93, 90, and 73%. Had the semiautomated system been used in 2010, a human would have needed to read only 605/5,616 citations to update the PDGene registry (11%) and 555/7,298 (8%), 717/5,381 (13%), and 334/1,015 (33%) for the other three databases. CONCLUSION: Data mining methodologies can reduce the burden of updating systematic reviews, without missing more papers than humans.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
This study and others that have evaluated semi-automated screening with support vector models, semantic vector models, and complement naïve Bayes models indicate that considerable workload savings can be achieved.
“This study and others that have evaluated semi-automated screening with support vector models [14, 15], semantic vector models [16], and complement naïve Bayes models [17] indicate that considerable workload savings can be achieved.”
From Faster title and abstract screening? Evaluating Abstrackr, a semi-automated online screening program for systematic reviewers (2015) · cites this paper as [15]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Systematic meta-analyses of Alzheimer disease genetic association studies: the AlzGene database
Bertram, L., McQueen, M. B., Mullin, K., Blacker, D., & Tanzi, R. E. (2007) · Nature Genetics · doi:10.1038/ng1934 · Cited by 1,857
Self-citationFu, R., Gartlehner, G., Grant, M., Shamliyan, T., Sedrakyan, A., Wilt, T. J., Griffith, L., Oremus, M., Raina, P., Ismaila, A., et al. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.08.010 · Cited by 730
Self-citationSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351
Self-citationEdwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., & Wentz, R. (2002) · Statistics in Medicine · doi:10.1002/sim.1190 · Cited by 340
Estimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163
Finding evidence for comparing medical interventions: AHRQ and the Effective Health Care Program
Relevo, R., & Balshem, H. (2011) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2010.11.022 · Cited by 52 · cited 1×
“32 36 Thus, computer-assisted screening could replace full manual screening for both screeners, replace one screener, or be used in addition to both screeners to further increase the sensitivity of the overall process.”
Method
Flagged first, then by citations
Research works printed but not linked: 12
- Institute of Medicine of the National Academies. Finding What Works in Healthcare. Standards for Systematic Reviews. National Academies Press: Washington, DC, 2011.
- Lill CM, Roehr JT, McQueen MB, et al. Comprehensive research synopsis and systematic meta-analyses in Parklinson’s disease: The PDGene database 2012, in press.
- Vapnik VN. The Nature of Statistical Learning Theory. Springer: New York, 2000. · DOI
- Joachims T. Text Categorization With Support Vector Machines: Learning With Many Relevant Features. Springer: Berlin, Germany, 1998. · DOI
- Van Hulse J, Khoshgoftaar TM, Napolitano A. Experimental perspectives on learning from imbalanced data. International Conference on Machine Learning (ICML) 2007:935–942. · DOI
- Small KM, Wallace BC, Brodley CE, Trikalinos TA. The constrained weight space SVM: Learning with labeled features. International Conference on Machine Learning (ICML) 2011:865–872.
- Wallace BC, Small KM, Brodley CE, Trikalinos TA. Active learning for biomedical citation screening. Knowledge Discovery and Data Mining (KDD) 2010:173–182. · DOI · Paper page
- Wallace BC, Small KM, Brodley CE, Trikalinos TA. Modeling annotation time to reduce workload in comparative effectiveness reviews. Proc ACM International Health Informatics Symposium (IHI) 2010:28–35. · DOI · Paper page
- Wallace BC, Small KM, Brodley CE, Trikalinos TA. Who should label what? Instance allocation in multiple expert active learning. Proc SIAM International Conference on Data Mining 2011:176–187. · DOI
- Breiman L. Bagging predictors. Journal of Machine Learning 1996:123–140. · DOI · Paper page
- Kang P, Cho S. EUS SVMs: ensemble of under-sampled SVMs for data imbalance problems 2006; 4232:837–846. · DOI
- Institute of Medicine. Finding What Works in Health Care. Standards for Systematic Reviews. National Acedemies Press: Washington, DC, 2011.
Cited by
Sorted by citations
Traumatic brain injury: integrated approaches to improve prevention, clinical care, and research
Maas, A. I. R., Menon, D. K., Adelson, P. D., Andelic, N., Bell, M. J., Belli, A., Bragge, P., Brazinova, A., Büki, A., Chesnut, R. M., et al. (2017) · The Lancet Neurology · doi:10.1016/s1474-4422(17)30371-x · Cited by 2,609
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Living Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596
Reducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Chai, K. E. K., Lines, R. L. J., Gucciardi, D. F., & Ng, L. (2021) · Systematic Reviews · doi:10.1186/s13643-021-01635-3 · Cited by 190
Rathbone, J., Hoffmann, T., & Glasziou, P. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0067-6 · Cited by 184
“Only limited research to date has been conducted into the strengths and limitations of semi-automated citation screening [14, 15].”
BackgroundResultBannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
“Publications discussing this approach include: Yu et al. (2008); Wallace et al. (2010c); Wallace et al. (2012a); and Wallace et al. (2012b).”
BackgroundMethodPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
“…machine learning into the systematic review toolkit has been showcased by numerous retrospective studies (simulations of the screening process using previously screened collections 9 , 13 , 15 - 31 ) and some prospective studies 32 , 33 that have explored different machine learning approaches.”
BackgroundMethodText mining applications in psychiatry: a systematic literature review
Abbe, A., Grouin, C., Zweigenbaum, P., & Falissard, B. (2016) · International Journal of Methods in Psychiatric Research · doi:10.1002/mpr.1481 · Cited by 130
A Machine Learning Approach for Semi-Automated Search and Selection in Literature Studies
Ros, R., Bjarnason, E., & Runeson, P. (2017) · Proceedings of the 21st International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/3084226.3084243 · Cited by 80
Olorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Olorisade, B. K., de Quincey, E., Brereton, P., & Andras, P. (2016) · Proceedings of the 20th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2915970.2915982 · Cited by 42
Metadata
- Authors
- Wallace, B. C., Small, K., Brodley, C. E., Lau, J., Schmid, C. H., Bertram, L., Lill, C. M., Cohen, J. T., & Trikalinos, T. A.
- Affiliations
- Tufts University (US)
- Tufts Medical Center (US)
- Max Planck Institute for Molecular Genetics (DE)
- Johannes Gutenberg University Mainz (DE)
- Corresponding author
- Wallace, Byron C.
- ORCID
- Venue
- Genetics in Medicine
- Publisher
- Elsevier BV
- Published
- 2012-07
- Volume
- 14
- Issue
- 7
- Pages
- 663-669
- ISSN
- 1098-3600 (ISSN-L)
- PMC
- PMC3908550
- OpenAlex
- W2035792132
- Cited by
- 62
- References
- 6
- Citations by year
- 2026: 2 · 2024: 5 · 2023: 3 · 2022: 2 · 2021: 6 · 2020: 7 · 2019: 5 · 2018: 5 · 2017: 4 · 2016: 5 · 2015: 8 · 2014: 6 · 2013: 2 · 2012: 2
- FWCI
- 1.59
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Genetics in Medicine (publishedVersion, cc-by-nc-nd)
- Europe PMC (PubMed Central) (publishedVersion, cc-by-nc-nd)
- PubMed Central (publishedVersion, cc-by-nc-nd)
- PubMed (publishedVersion)
- MPG.PuRe (Max Planck Society) (submittedVersion, other-oa)
- Funders
- Agency for Healthcare Research and Quality (R01HS018494)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- systematic review updating, citation screening, data mining, classification models, Parkinson's disease, Alzheimer's disease, schizophrenia, cost-effectiveness analysis, MEDLINE, literature search automation
- MeSH terms
- Systematic Reviews as Topic, Alzheimer Disease, Cost-Benefit Analysis, Humans, Parkinson Disease, Periodicals as Topic, Schizophrenia, Software, Technology Assessment, Biomedical, Meta-Analysis as Topic, Databases, Factual, Empirical Research, Data Mining
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
