Supporting systematic reviews using LDA-based document representations
Open accessThe DOI matches its Crossref record; 11 of the 11 references it cites were found in a registry or an index.
Abstract
BACKGROUND: Identifying relevant studies for inclusion in a systematic review (i.e. screening) is a complex, laborious and expensive task. Recently, a number of studies has shown that the use of machine learning and text mining methods to automatically identify relevant studies has the potential to drastically decrease the workload involved in the screening phase. The vast majority of these machine learning methods exploit the same underlying principle, i.e. a study is modelled as a bag-of-words (BOW). METHODS: We explore the use of topic modelling methods to derive a more informative representation of studies. We apply Latent Dirichlet allocation (LDA), an unsupervised topic modelling approach, to automatically identify topics in a collection of studies. We then represent each study as a distribution of LDA topics. Additionally, we enrich topics derived using LDA with multi-word terms identified by using an automatic term recognition (ATR) tool. For evaluation purposes, we carry out automatic identification of relevant studies using support vector machine (SVM)-based classifiers that employ both our novel topic-based representation and the BOW representation. RESULTS: Our results show that the SVM classifier is able to identify a greater number of relevant studies when using the LDA representation than the BOW representation. These observations hold for two systematic reviews of the clinical domain and three reviews of the social science domain. CONCLUSIONS: A topic-based feature representation of documents outperforms the BOW representation when applied to the task of automatic citation screening. The proposed term-enriched topics are more informative and less ambiguous to systematic reviewers.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681
Self-citationFormulating Questions and Locating Primary Studies for Inclusion in Systematic Reviews
Counsell, C. (1997) · Annals of Internal Medicine · doi:10.7326/0003-4819-127-5-199709010-00008 · Cited by 505 · cited 1×
“Background The screening phase of systematic reviews aims to identify citations relevant to a research topic, according to a certain pre-defined protocol [1–4] known as the Population, the Intervention, the Comparator and the Outcome (PICO) framework.”
BackgroundReducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · cited 1×
“Cohen et al. [33] constructed a voting perceptron-based automated citation classification system which is able to reduce the number of articles that needs to be reviewed by more than 50 %.”
BackgroundSemi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · cited 2×
“To overcome these limitations, methods such as machine learning, text mining [9, 10], text classification [11] and active learning [6, 12] have been used to partially automate this process, in order to reduce the workload, without sacrificing the quality of the reviews.”
BackgroundMethodReducing systematic review workload through certainty-based screening
Miwa, M., Thomas, J., O’Mara-Eves, A., & Ananiadou, S. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.06.005 · Cited by 190
Self-citationCombining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
Automatic text classification to support systematic reviews in medicine
García Adeva, J. J., Pikatza Atxa, J. M., Ubeda Carrillo, M., & Ansuategi Zengotitabengoa, E. (2014) · Expert Systems with Applications · doi:10.1016/j.eswa.2013.08.047 · Cited by 125 · cited 1×
“To overcome these limitations, methods such as machine learning, text mining [9, 10], text classification [11] and active learning [6, 12] have been used to partially automate this process, in order to reduce the workload, without sacrificing the quality of the reviews.”
MethodActive learning for biomedical citation screening
Wallace, B. C., Small, K., Brodley, C. E., & Trikalinos, T. A. (2010) · Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1835804.1835829 · Cited by 123
A new algorithm for reducing the workload of experts in performing systematic reviews
Matwin, S., Kouznetsov, A., Inkpen, D., Frunza, O., & O'Blenis, P. (2010) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.2010.004325 · Cited by 100 · cited 1×
“Matwin et al. [32] exploited a factorised complement naive Bayes classifier for reducing the workload of experts reviewing journal articles for building systematic reviews of drug class efficacy.”
BackgroundScreening nonrandomized studies for medical systematic reviews: a comparative study of classifiers
Bekhuis, T., & Demner-Fushman, D. (2012) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2012.05.002 · Cited by 70 · cited 1×
“Bekhuis et al. [34] investigated the performance of different classifiers and feature sets in terms of their ability to reduce workload.”
Background
Research works printed but not linked: 26
- Barza M, Trikalinos TA, Lau J. Statistical considerations in meta-analysis. Infect Dis Clin North Am. 2009; 23(2):195–210. · DOI
- Demner-Fushman D, Lin J. Answering clinical questions with knowledge-based and statistical techniques. Comput Ling. 2007; 33(1):63–103. · DOI · Paper page
- Hunter L, Cohen KB. Biomedical language processing: perspective what’s beyond PubMed?Mol Cell. 2006; 21(5):589. · DOI · Paper page
- Cohen AM. Performance of support-vector-machine-based classification on 15 systematic review topics evaluated with the WSS@95 measure. J Am Med Inform Assoc. 2011; 18(1):104–4. · DOI · Paper page
- O’Mara-Eves A, Thomas J, McNaught J, Miwa M, Ananiadou S. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Sys Rev. 2015; 4(1). doi: 10.1186/2046-4053-4-5 . Highly Accessed. · DOI · Paper page
- Lukins SK, Kraft NA, Etzkorn LH. Source code retrieval for bug localization using latent Dirichlet allocation. In: Reverse Engineering, 2008. WCRE’08. 15th Working Conference On. Antwerp: IEEE: 2008. p. 155–64.
- Henderson K, Eliassi-Rad T. Applying latent Dirichlet allocation to group discovery in large graphs. In: Proceedings of the 2009 ACM Symposium on Applied Computing. New York, NY, USA: ACM: 2009. p. 1456–1461.
- Maskeri G, Sarkar S, Heafield K. Mining business topics in source code using latent Dirichlet allocation. In: Proceedings of the 1st India Software Engineering Conference. New York, NY, USA: ACM: 2008. p. 113–20.
- Linstead E, Lopes C, Baldi P. An application of latent Dirichlet allocation to analyzing software evolution. In: Machine learning and applications, 2008. ICMLA’08. Seventh International Conference On. San Diego, CA: IEEE: 2008. p. 813–8.
- Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. Js Mach Learn Res. 2003; 3:993–1022.
- Redner RA, Walker HF. Mixture densities, maximum likelihood and the EM algorithm. SIAM Rev. 1984; 26(2):195–239. · DOI · Paper page
- Berry MW, Dumais ST, O’Brien GW. Using linear algebra for intelligent information retrieval. SIAM review. 1995; 37(4):573–595. · DOI · Paper page
- Lee M, Mimno D. Low-dimensional embeddings for interpretable anchor-based topic inference. In: Proceedings of Empirical Methods in Natural Language Processing: 2014. · DOI
- Arora S, Ge R, Moitra A. Learning topic models—going beyond SVD. In: Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium On. New Brunswick, NJ: IEEE: 2012. p. 1–10.
- Bekhuis T, Demner-Fushman D. Towards automating the initial screening phase of a systematic review. Stud Health Technol Inform. 2010; 160(Pt 1):146–50.
- Octaviano FR, Felizardo KR, Maldonado JC, Fabbri SC. Semi-automatic selection of primary studies in systematic literature reviews: is it reasonable?Empir Softw Eng. 2015; 20(6):1898–1917. · DOI
- Cohen AM. Optimizing feature representation for automated systematic review work prioritization. In: AMIA Annual Symposium Proceedings: 2008. p. 121. American Medical Informatics Association.
- Romero Felizardo K, Souza SR, Maldonado JC. The use of visual text mining to support the study selection activity in systematic literature reviews: a replication study. In: Replication in empirical software engineering research (RESER), 2013 3rd International Workshop On. Baltimore, MD: IEEE: 2013. p. 91–100.
- Fiszman M, Bray BE, Shin D, Kilicoglu H, Bennett GC, Bodenreider O, et al.Combining relevance assignment with quality of the evidence to support guideline development. Stud Health Technol Inform. 2010; 160(01):709.
- Frunza O, Inkpen D, Matwin S. Building systematic reviews using automatic text classification techniques. In: Proceedings of the 23rd International Conference on Computational Linguistics: Posters. Beijing, China: Association for Computational Linguistics: 2010. p. 303–11.
- Jonnalagadda S, Petitti D. A new iterative method to reduce workload in systematic review process. Int J Comput Biol Drug Des. 2013; 6(1):5–17. · DOI · Paper page
- Frantzi K, Ananiadou S, Mima H. Automatic recognition of multi-word terms: the C-value/NC-value method. Int J Digit Libr. 2000; 3(2):115–30. · DOI · Paper page
- Chang CC, Lin CJ. LIBSVM: a library for support vector machines. ACM Trans Intell Syst Technol (TIST). 2011; 2(3):27.
- Řehuřek R, Sojka P. Software framework for topic modelling with large corpora. In: Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks. Valletta, Malta: ELRA: 2010. p. 45–50. http://is.muni.cz/publication/884893/en .
- Kotsiantis S, Kanellopoulos D, Pintelas P. Handling imbalanced datasets: a review. GESTS Int Trans Comput Sci Eng. 2006; 30(1):25–36.
- Akbani R, Kwek S, Japkowicz N. Applying support vector machines to imbalanced datasets. In: Machine learning: ECML 2004. Berlin Heidelbeg: Springer: 2004. p. 39–50.
Cited by
Marshall, I. J., & Wallace, B. C. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1074-9 · Cited by 638
Howard, B. E., Phillips, J., Tandon, A., Maharana, A., Elmore, R., Mav, D., Sedykh, A., Thayer, K., Merrick, B. A., Walker, V., et al. (2020) · Environment International · doi:10.1016/j.envint.2020.105623 · Cited by 157
“Variants of this active learning approach for document screening have been tested previously in several research scenarios (Wallace et al., 2010a, 2010b; Wallace and Small, 2011; Wallace et al., 2012a; Miwa et al., 2014; Mo et al., 2015; Rathbone et al., 2015).”
MethodOlorisade, B. K., Brereton, P., & Andras, P. (2017) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2017.07.010 · Cited by 54
Machine learning techniques for the automation of literature reviews and systematic reviews in EFSA
Jaspers, S., De Troyer, E., & Aerts, M. (2018) · EFSA Supporting Publications · doi:10.2903/sp.efsa.2018.en-1427 · Cited by 36
“The authors noted that these results were in line with the findings of Qi et al. (2013) and Rathbone et al. (2015). More specifically, these authors argued that the automatic deduplication option in EndNote is not fully adequate and should be supplemented by a manual search.”
ResultInfluentialDiscriminating between empirical studies and nonempirical works using automated text classification
Langlois, A., Nie, J., Thomas, J., Hong, Q. N., & Pluye, P. (2018) · Research Synthesis Methods · doi:10.1002/jrsm.1317 · Cited by 26
Metadata
- Authors
- Mo, Y., Kontonatsios, G., & Ananiadou, S.
- Venue
- Systematic Reviews
- Published
- 2015-11-26
- Volume
- 4
- Issue
- 1
- Pages
- 172-172
- ISSN
- 2046-4053 (ISSN-L)
- OpenAlex
- W2180673283
- Cited by
- 83
- References
- 11
- FWCI
- 0.82
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Supporting systematic reviews using LDA-based document representations. https://app.cochat.ai/papers/10.1186/s13643-015-0117-0
- The paper
- Mo, Y., Kontonatsios, G., & Ananiadou, S. (2015). Supporting systematic reviews using LDA-based document representations. Systematic Reviews. https://doi.org/10.1186/s13643-015-0117-0
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.