Improving Active Learning in Systematic Reviews
Open accessAbstract
Systematic reviews are essential to summarizing the results of different clinical and social science studies. The first step in a systematic review task is to identify all the studies relevant to the review. The task of identifying relevant studies for a given systematic review is usually performed manually, and as a result, involves substantial amounts of expensive human resource. Lately, there have been some attempts to reduce this manual effort using active learning. In this work, we build upon some such existing techniques, and validate by experimenting on a larger and comprehensive dataset than has been attempted until now. Our experiments provide insights on the use of different feature extraction models for different disciplines. More importantly, we identify that a naive active learning based screening process is biased in favour of selecting similar documents. We aimed to improve the performance of the screening process using a novel active learning algorithm with success. Additionally, we propose a mechanism to choose the best feature extraction method for a given review.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The application of active learning models in systematic reviews has been extensively studied in simulation studies.
“The application of active learning models in systematic reviews has been extensively studied in simulation studies [11-21].”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Reducing Workload in Systematic Review Preparation Using Automated Citation Classification
Cohen, A. M., Hersh, W. R., Peterson, K., & Yen, P.-Y. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1929 · Cited by 411 · see citing works
Semi-automated screening of biomedical citations for systematic reviews
Wallace, B. C., Trikalinos, T. A., Lau, J., Brodley, C., & Schmid, C. H. (2010) · BMC Bioinformatics · doi:10.1186/1471-2105-11-55 · Cited by 351 · see citing works
Flagged first, then by citations
Research works printed but not linked: 11
- [1][1] C. C. Beahler, J. J. Sundheim, and N. I. Trapp. Information retrieval in systematic reviews: challenges in the public health arena. American journal of preventive medicine , 18(4):6-10, 2000. -
- [2][2] D. M. Blei, A. Y. Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of machine Learning research , 3(Jan):993-1022, 2003. -
- [3][3] I. Chalmers, L. V. Hedges, and H. Cooper. A brief history of research synthesis. Evaluation & the health professions , 25(1):12-37, 2002. -
- [5][5] A. M. Dai, C. Olah, and Q. V. Le. Document embedding with paragraph vectors. arXiv preprint arXiv:1507.07998 , 2015. - · arXiv
- [7][7] K. Hashimoto, G. Kontonatsios, M. Miwa, and S. Ananiadou. Topic detection using paragraph vectors to support active learning in systematic reviews. Journal of Biomedical Informatics , 2016. -
- [8][8] Q. V. Le and T. Mikolov. Distributed representations of sentences and documents. In ICML , volume 14, pages 1188-1196, 2014. -
- [9][9] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, and J. Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems , pages 3111-3119, 2013. -
- [10][10] M. Miwa, J. Thomas, A. O'Mara-Eves, and S. Ananiadou. Reducing systematic review workload through certainty-based screening. Journal of biomedical informatics , 51:242-253, 2014. -
- [11][11] Y. Mo, G. Kontonatsios, and S. Ananiadou. Supporting systematic reviews using lda-based document representations. Systematic reviews , 4(1):1, 2015. -
- [12][12] A. O'Mara-Eves, J. Thomas, J. McNaught, M. Miwa, and S. Ananiadou. Using text mining for study identification in systematic reviews: a systematic review of current approaches. Systematic reviews , 4(1):1, 2015. -
- [14][14] S. Tong and D. Koller. Support vector machine active learning with applications to text classification. Journal of machine learning research , 2(Nov):45-66, 2001. -
Cited by
Sorted by citations
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., Teijema, J. J., & van de Schoot, R. (2023) · Systematic Reviews · doi:10.1186/s13643-023-02257-7 · Cited by 96 · see citing works
Active learning for screening prioritization in systematic reviews - A simulation study
Ferdinands, G., Schram, R., de Bruin, J., Bagheri, A., Oberski, D. L., Tummers, L., & van de Schoot, R. (2020) · doi:10.31219/osf.io/w6qbg · Cited by 51 · see citing works
- Authors
- Singh, G., Thomas, J., & Shawe-Taylor, J.
- Affiliations
- University College London (GB)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2018
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1801.09496
- OpenAlex
- W2785540181
- Cited by
- 11
- References
- 2
- Citations by year
- 2023: 3 · 2022: 1 · 2021: 2 · 2020: 2 · 2019: 2 · 2018: 1
- FWCI
- 0.35
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Medical Research Council (MR/J005037/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning and Algorithms
- Keywords
- active learning (machine learning), systematic reviews, literature screening, study selection, feature extraction, document similarity, information retrieval
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
