Advanced analytics for the automation of medical systematic reviews
The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“When using cross-validation, 11 studies adopted the 10-fold or 5 × 2-fold approach [33,37,38,43,44,48,52,61,63,66,67].”
From Automation of systematic literature reviews: A systematic literature review (2021) · cites this paper as [43]
Citations
Cites
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421
Cortes, C., & Vapnik, V. (1995) · Machine-mediated learning · doi:10.1023/a:1022627411411 · Cited by 48,445 · cited 1×
“That is, if the C value is large, than model is better fitted to the training data (may cause over-fitting), whereas if the C value is small, SVM fits on the bulk of data (Cortes and Vapnik 1995).”
BackgroundCochrane Handbook for Systematic Reviews of Interventions: Cochrane Book Series
(2008) · doi:10.1002/9780470712184 · Cited by 25,048
Text categorization with Support Vector Machines: Learning with many relevant features
Joachims, T. (1998) · Lecture Notes in Computer Science · doi:10.1007/bfb0026683 · Cited by 7,999 · cited 2×
“Existing studies such as (Joachims 1998; Liu et al. 2002; Bekhuis and Demner-Fushman 2012) has proved the effectiveness of SVM with a linear kernel in text classification in the process of medical systematic reviews.”
BackgroundApproximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
Dietterich, T. G. (1998) · Neural Computation · doi:10.1162/089976698300017197 · Cited by 3,840 · cited 1×
“Then the roles of the two half data sets are exchanged, with the second half used for training and the first half used for testing, with the results accumulated from both halves of the split (Dietterich 1998).”
MethodExploratory Undersampling for Class-Imbalance Learning
Liu, X.-Y., Wu, J., & Zhou, Z.-H. (2009) · IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) · doi:10.1109/tsmcb.2008.2007853 · Cited by 2,465
Systematic Reviews: Rationale for systematic reviews
Mulrow, C. D. (1994) · BMJ · doi:10.1136/bmj.309.6954.597 · Cited by 1,677
Understanding inverse document frequency: on theoretical arguments for IDF
Robertson, S. (2004) · Journal of Documentation · doi:10.1108/00220410410560582 · Cited by 1,541 · cited 1×
“Following (Bekhuis and Demner-Fushman 2012), we used the method TF-IDF(term frequency/inverse document frequency) (Robertson 2004) to create the weights.”
MethodData Mining for Imbalanced Datasets: An Overview
Chawla, N. V. (2009) · Data Mining and Knowledge Discovery Handbook · doi:10.1007/978-0-387-09823-4_45 · Cited by 1,082 · cited 6×
“A particular type of oversampling, namely the Synthetic Minority Oversampling Technique (SMOTE) (Chawla et al., 2010 Overall, the findings of extant research show enough promise to further consider the possibility of using data analytics techniques for automatically screening articles for…”
BackgroundMethodInfluentialImbalanced Learning: Foundations, Algorithms, and Applications
(2013) · Wiley eBooks · doi:10.1002/9781118646106 · Cited by 831 · cited 1×
“Oversampling seeks to increase the number of samples in the minority class by replicating samples from that class (He and Ma 2013).”
BackgroundSystematic review automation technologies
Tsafnat, G., Glasziou, P., Choong, M. K., Dunn, A., Galgani, F., & Coiera, E. (2014) · Systematic Reviews · doi:10.1186/2046-4053-3-74 · Cited by 472 · cited 1×
“…(Chawla et al., 2010 Overall, the findings of extant research show enough promise to further consider the possibility of using data analytics techniques for automatically screening articles for systematic reviews (Cohen et al. 2006; Frunza et al. 2010; Shemilt et al. 2013; Tsafnat et al. 2014).”
BackgroundLETOR: Benchmark Dataset for Research on Learning to Rank for Information Retrieval
Liu, T., Xu, J., Qin, T., Xiong, W., & Li, H. (2007) · Cited by 419
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 17×
“…the Unified Medical Language Systems (UMLS) to extract medical terms as features for article classification, while the majority of existing research uses the “ bag-of-words ” approach (Adeva et al. 2014; Shemilt et al. 2013; Bekhuis and Demner-Fushman 2012; Wallace et al. 2010; Cohen et al. 2006).”
BackgroundMethodResultInfluentialSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 1×
“…the Unified Medical Language Systems (UMLS) to extract medical terms as features for article classification, while the majority of existing research uses the “ bag-of-words ” approach (Adeva et al. 2014; Shemilt et al. 2013; Bekhuis and Demner-Fushman 2012; Wallace et al. 2010; Cohen et al. 2006).”
MethodSystematic reviews need systematic searchers.
(2005) · Europe PMC (PubMed Central) · Cited by 257
The Unified Medical Language System (UMLS) of the National Library of Medicine.
Lindberg, C. H. (1990) · PubMed · Cited by 207
Shemilt, I., Simon, A., Hollands, G. J., Marteau, T. M., Ogilvie, D., O'Mara‐Eves, A., Kelly, M. P., & Thomas, J. (2014) · Research Synthesis Methods · doi:10.1002/jrsm.1093 · Cited by 170
Estimating Time to Conduct a Meta-analysis From Number of Citations Retrieved
Allen, I. E. (1999) · JAMA: The Journal of the American Medical Association · doi:10.1001/jama.282.7.634 · Cited by 163 · cited 1×
“…manual workflow of selecting articles for systematic reviews (SRs), developing SRs requires a significant investment in time (1139 expert hours on average) and funds (up to a quarter of a million dollars) from a dedicated and qualified research team (Allen and Olkin 1999; McGowan and Sampson 2005).”
BackgroundSupporting Systematic Reviews Using Text Mining
Ananiadou, S., Rea, B., Okazaki, N., Procter, R., & Thomas, J. (2009) · Social Science Computer Review · doi:10.1177/0894439309332293 · Cited by 144 · cited 1×
“There have been some attempts in literature to leverage analytics to automate systematic reviewer generation and update (Ananiadou et al. 2009; Bekhuis and Demner-Fushman 2012; Cohen et al.; Frunza et al. 2010; Shemilt et al. 2013).”
BackgroundAutomatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 4×
“There are efforts that have leveraged text analytics (Bekhuis and Demner-Fushman 2012; Shemilt et al. 2013; Adeva et al. 2014) to automate the article screening procedure for systematic reviews.”
BackgroundMethodInfluentialLiu, H. (2002) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1101 · Cited by 114 · cited 2×
“Prior research (Liu et al. 2002; Aronson et al. 2007), though not in the area of systematic reviews, has corroborated the observation that biomedical text classification can be improved by enriching raw text with automatically extracted Unified Medical Language System (UMLS) terms.”
BackgroundSystematic Reviews: The Heart of Evidence-based Practice
Stevens, K. R. (2001) · AACN Clinical Issues: Advanced Practice in Acute and Critical Care · doi:10.1097/00044067-200111000-00009 · Cited by 113 · cited 1×
“This research focuses on systematic reviews, the heart of evidence-based medical practice (Stevens 2001).”
BackgroundTowards Automatic Recognition of Scientifically Rigorous Clinical Research Evidence
Kilicoglu, H., Demner-Fushman, D., Rindflesch, T. C., Wilczynski, N. L., & Haynes, R. B. (2009) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2996 · Cited by 101 · cited 1×
“As an example, Kilicoglu et al. (2009) demonstrated the feasibility of automatically identifying “ scientifically rigorous ” articles using multiple features from publications, including “ high-level ” features such as Unified Medical Language System (UMLS) terms.”
MethodA new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · cited 2×
“Any system with a low recall would be of little use (Matwin et al. 2010).”
BackgroundThe effect of oversampling and undersampling on classifying imbalanced text datasets
Liu, A. Y.-C. (2004) · doi:10.26153/tsw/12300 · Cited by 91
Screening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 13×
“(UMLS) to extract medical terms as features for article classification, while the majority of existing research uses the “bagof-words” approach (Adeva et al. 2014; Shemilt et al. 2013; Bekhuis and Demner-Fushman 2012; Wallace et al. 2010; Cohen et al. 2006).”
BackgroundMethodResultInfluentialAronson, A. R., Bodenreider, O., Demner-Fushman, D., Fung, K. W., Lee, V. K., Mork, J. G., Névéol, A., Peters, L., & Rogers, W. J. (2007) · Proceedings of the Workshop on BioNLP 2007 Biological, Translational, and Clinical Language Processing - BioNLP '07 · doi:10.3115/1572392.1572412 · Cited by 54 · cited 1×
“Prior research (Liu et al. 2002; Aronson et al. 2007), though not in the area of systematic reviews, has corroborated the observation that biomedical text classification can be improved by enriching raw text with automatically extracted Unified Medical Language System (UMLS) terms.”
BackgroundShojania, K. G., Sampson, M., Ansari, M. T., Ji, J., Garritty, C., Rader, T., & Moher, D. (2007) · Europe PMC (PubMed Central) · Cited by 51
The perceptron algorithm vs. Winnow
Kivinen, J., & Warmuth, M. K. (1995) · Conference on Learning Theory · doi:10.1145/225298.225333 · Cited by 49
Cohen, A. M., Adams, C. E., Davis, J. M., Yu, C., Yu, P. S., Meng, W., Duggan, L., McDonagh, M., & Smalheiser, N. R. (2010) · Proceedings of the 1st ACM International Health Informatics Symposium · doi:10.1145/1882992.1883046 · Cited by 45 · cited 1×
“Evidence Based Medicine (EBM) refers to the application of state-of-the-art medical evidence to improve the quality and reduce the cost of medical care (Cohen et al. 2010).”
Background(2010) · Cited by 32
Role of Information Technology in Evidence Based Medicine: Advantages and Limitations
(2007) · The Internet Journal of Healthcare Administration · doi:10.5580/7c9 · Cited by 11 · cited 1×
“Information technology (IT) plays a crucial role in the practice of evidence-based medicine (EBM) by allowing health care practitioners to access and evaluate clinical evidence as they formulate their patient care strategies (Wells 2006).”
BackgroundSupporting Systematic Reviews using Text Mining
Ananiadou, S., Procter, R., Rea, B., Sasaki, Y., & Thomas, J. (2007) · Research Explorer (The University of Manchester) · Cited by 6
(2011) · Medical Entomology and Zoology
Research works printed but not linked: 1
- Research Support, U.S. Gov't, P.H.S.]. J Am Med Inform Assoc, 9(6), 621–636.
Cited by
Automation of systematic literature reviews: A systematic literature review
van Dinter, R., Tekinerdogan, B., & Catal, C. (2021) · Information and Software Technology · doi:10.1016/j.infsof.2021.106589 · Cited by 432
“…and Saquicela [40] describe an algorithm that automates metadata extraction from articles Cost assignment [66] Data resampling [41,42,58,64] SMOTE [43,53] Feature enrichment [43,63] Cost-sensitive classifiers [43,53, 61,66] Precision@95% recall [ published by IEEE, ACM, Springer, Scopus, and…”
MethodCan ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275
“If there are, in fact, clusters of relevant documents amongst a large number of non-relevant ones, then automated classification techniques may help to uncover these [2, 32, 41, 70, 74].”
BackgroundPrioritising references for systematic reviews with RobotAnalyst: A user study
Przybyła, P., Brockmeier, A. J., Kontonatsios, G., Le Pogam, M., McNaught, J., von Elm, E., Nolan, K., & Ananiadou, S. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1311 · Cited by 148
Big data and analytics in healthcare: Introduction to the special section
Kankanhalli, A., Hahn, J., Tan, S. S.-L., & Gao, G. (2016) · Information Systems Frontiers · doi:10.1007/s10796-016-9641-2 · Cited by 180
“The first paper by Timsina et al. (2016) provides an advanced analytics solution for automation of medical systematic reviews.”
BackgroundOfori-Boateng, R., Aceves-Martins, M., Wiratunga, N., & Moreno-Garcia, C. F. (2024) · Artificial Intelligence Review · doi:10.1007/s10462-024-10844-w · Cited by 86
Metadata
- Authors
- Timsina, P., Liu, J., & El-Gayar, O.
- Affiliations
- Dakota State University (US)
- Corresponding author
- Timsina, Prem
- Venue
- Information Systems Frontiers
- Published
- 2015-08-18
- Volume
- 18
- Issue
- 2
- Pages
- 237-252
- ISSN
- 1387-3326 (ISSN-L)
- 1572-9419
- OpenAlex
- W1203019046
- Cited by
- 33
- References
- 34
- Citations by year
- 2025: 1 · 2024: 3 · 2023: 4 · 2022: 1 · 2021: 7 · 2020: 5 · 2019: 3 · 2018: 3 · 2017: 2 · 2016: 4
- FWCI
- 0.50
- Type
- journal article
- Language
- en
- Open access
- closed
- Licence
- www.springer.com/tdm (tdm)
- Locations
- Information Systems Frontiers (publishedVersion)
- RePEc: Research Papers in Economics (submittedVersion)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Keywords
- systematic reviews, UMLS, class imbalance, resampling, evidence-based medicine
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.