Automation of citation screening in pre-clinical systematic reviews
Open accessThe DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
Background: The amount of published in vivo studies and the speed researchers are publishing them make it virtually impossible to follow the recent development in the field. Systematic review emerged as a method to summarise and analyse the studies quantitatively and critically but it is often out-of-date due to its lengthy process. Method: We invited five machine learning and text-mining groups to build classifiers for identifying publications relevant to neuropathic pain (33814 training publications). We kept 1188 publications for the assessment of the performance of different classifiers. Two groups participated in the next stage: testing their algorithm on datasets labeled for psychosis (11777/2944) and datasets labeled for Vitamin D in multiple sclerosis (train/text: 2038/510). Result: The performances (sensitive/specificity) of the most promising classifier built for neuropathic pain are: 95%/84%. The performance for psychosis and Vitamin D in multiple sclerosis datasets are 95%/73% and 100%/45%. Conclusions: Machine learning can significantly reduce the irrelevant publications in a systematic review, and save the scientists’ time and money. Classifier algorithms built for one dataset can be reapplied on another dataset in different field. We are building a machine learning service at the back of Systematic Review & Meta-analysis Facility (SyRF).
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“We chose an inclusion cut point which gave 99% precision and obtained 50,653 abstracts describing in vivo animal experiments.”
From PICO entity extraction for preclinical animal literature. (2022) · cites this paper as [16]
Citations
Cites
O’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/2046-4053-4-5 · Cited by 681 · cited 1×
“Text-mining has previously been tested, used and analysed in clinical systematic review area [4].”
MethodSelf-citationLiving Systematic Reviews: An Emerging Opportunity to Narrow the Evidence-Practice Gap
Elliott, J. H., Turner, T., Clavisi, O., Thomas, J., Higgins, J. P. T., Mavergames, C., & Gruen, R. L. (2014) · PLoS Medicine · doi:10.1371/journal.pmed.1001603 · Cited by 596
Self-citationO’Mara-Eves, A., Thomas, J., McNaught, J., Miwa, M., & Ananiadou, S. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0031-5 · Cited by 353
Self-citationSena, E. S., Currie, G. L., McCann, S. K., Macleod, M. R., & Howells, D. W. (2014) · Journal of Cerebral Blood Flow & Metabolism · doi:10.1038/jcbfm.2014.28 · Cited by 264
Self-citationApplications of text mining within systematic reviews
Thomas, J., McNaught, J., & Ananiadou, S. (2011) · Research Synthesis Methods · doi:10.1002/jrsm.27 · Cited by 212
Self-citationSWIFT-Review: a text-mining workbench for systematic review
Howard, B. E., Phillips, J., Miller, K., Tandon, A., Mav, D., Shah, M. R., Holmgren, S., Pelch, K. E., Walker, V., Rooney, A. A., et al. (2016) · Systematic Reviews · doi:10.1186/s13643-016-0263-z · Cited by 196 · cited 2×
“(NaCTeM), Manchester UK; SWIFT [8], DoCTER [7] and our in house CAMARADES group at the University of”
BackgroundSelf-citationVarghese, A., Cawley, M. A., & Hong, T. (2017) · Environment Systems & Decisions · doi:10.1007/s10669-017-9670-5 · Cited by 35 · cited 2×
“(NaCTeM), Manchester UK; SWIFT [8], DoCTER [7] and our in house CAMARADES group at the University of”
BackgroundSelf-citationSearch For Studies Using Animal Models Of Neuropathic Pain
Currie, G. L., Sherratt, N., Colvin, L., Rice, A. S. C., Cramond, F., Macleod, M. R., & Sena, E. S. (2015) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.35448 · Cited by 1
Self-citationLiao, J. (2018) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.1194728 · Cited by 1
Self-citation
Research works printed but not linked: 1
- Gillian L Currie , Nicki Sherratt , Lesley A Colvin , Andrew SC Rice , Fala Cramond , Malcolm R Macleod , & Emily S Sena (2015). Systematic review and meta-analysis of studies using animal models of neuropathic pain protocol https://drive.google.com/file/d/0B30wfjnG6aEQRS0yMVNLWmdKeDQ/view
Cited by
Bannach-Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2019) · Systematic Reviews · doi:10.1186/s13643-019-0942-7 · Cited by 185
“To the best of our knowledge, there have been only two attempts to apply such techniques to reviews of preclinical animal studies [3, 19].”
BackgroundSelf-citationCurrie, G. L., Angel-Scott, H. N., Colvin, L., Cramond, F., Hair, K., Khandoker, L., Liao, J., Macleod, M. R., McCann, S., Morland, R., et al. (2019) · PLoS Biology · doi:10.1371/journal.pbio.3000243 · Cited by 87
“Five machine learning groups participated, and 13 classifiers were created and applied to the updated search (validation set) [16].”
MethodSelf-citationBahor, Z., Liao, J., Currie, G. L., Ayder, C., Macleod, M. R., McCann, S., Bannach‐Brown, A., Wever, K. E., Soliman, N., Wang, Q., et al. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100103 · Cited by 81
Self-citationHanegraaf, P., Wondimu, A., Mosselman, J. J., de Jong, R., Abògúnr̀in, Ṣ., Queiros, L., Lane, M., Postma, M. J., Boersma, C., & van der Schans, J. (2024) · BMJ Open · doi:10.1136/bmjopen-2023-076912 · Cited by 69
PICO entity extraction for preclinical animal literature.
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022) · Systematic Reviews · doi:10.1186/s13643-022-02074-4 · Cited by 29
“We downloaded 2,207,654 articles from the PubMed Central Open Access Subset database 1 published from 2010 to 2019 and used a citation screening filter trained to identify in vivo research from title and abstract (developed by EPPI-Centre, UCL [16]).”
MethodSelf-citationTechnological advances in preclinical meta-research
Bannach‐Brown, A., Hair, K., Bahor, Z., Soliman, N., Macleod, M. R., & Liao, J. (2021) · BMJ Open Science · doi:10.1136/bmjos-2020-100131 · Cited by 28
Self-citationSoliman, N., Hohmann, A. G., Haroutounian, S., Wever, K. E., Rice, A. S. C., & Finn, D. P. (2019) · PAIN Reports · doi:10.1097/pr9.0000000000000766 · Cited by 23
Self-citationBannach‐Brown, A., Przybyła, P., Thomas, J., Rice, A. S. C., Ananiadou, S., Liao, J., & Macleod, M. R. (2018) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/255760 · Cited by 24
“To the best of our knowledge there have been only two attempts to apply such 94 techniques to reviews of preclinical animal studies [3; 19].”
MethodSelf-citationSearching for evidence in public health emergencies: a white paper of best practices
Brody, S., Loree, S., Sampson, M., Mensinkai, S., Coffman, J. L., Mueller, M., Askin, N., Hamill, C., Wilson, E., McAteer, M. B., et al. (2023) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2023.1530 · Cited by 8
Rich, A. J., McGorray, E. L., Baldwin-SoRelle, C., Cawley, M. A., Grigg, K., Beach, L. B., Phillips, G., & Poteat, T. (2025) · Journal of the Medical Library Association JMLA · doi:10.5195/jmla.2025.1860 · Cited by 5
“Consensus is that a recall threshold of 95% is an acceptable level of risk for systematic reviews using AI-assisted screening methodology [27, 28].”
Wu, L., Chen, S., Guo, L., Shpyleva, S., Harris, K., Fahmi, T., Flanigan, T., Tong, W., Xu, J., & Ren, Z. X. (2022) · Regulatory Toxicology and Pharmacology · doi:10.1016/j.yrtph.2022.105287 · Cited by 9
Building a Systematic Online Living Evidence Summary of COVID-19 Research
Hair, K., Sena, E. S., Wilson, E., Currie, G. L., Macleod, M. R., Bahor, Z., Sena, C., Ayder, C., Liao, J., Tanriver-Ayder, E., et al. (2021) · Journal of EAHIL · doi:10.32384/jeahil17465 · Cited by 4
Self-citationFarquhar-Smith, P. W., & Gordon-Williams, R. (2020) · Faculty Opinions – Post-Publication Peer Review of the Biomedical Literature · doi:10.3410/f.735799554.793571725
Metadata
- Authors
- Liao, J., Ananiadou, S., Currie, L. G., Howard, B. E., Rice, A., Sena, S. E., Thomas, J., Varghese, A., & Macleod, M. R.
- Affiliations
- University of Edinburgh (GB)
- University of Manchester (GB)
- Imperial College London (GB)
- Corresponding author
- Liao, J.
- Venue
- bioRxiv (Cold Spring Harbor Laboratory)
- Publisher
- openRxiv
- Published
- 2018-03-12
- OpenAlex
- W2792585084
- Cited by
- 13
- References
- 9
- Citations by year
- 2025: 1 · 2024: 1 · 2023: 1 · 2022: 2 · 2021: 3 · 2020: 1 · 2019: 3 · 2018: 1
- Type
- posted content
- Language
- en
- Open access
- green
- doi.org
- Licence
- cc-by-nc-nd
- Locations
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
- Funders
- Wellcome Trust (MR/N015665/1)
- Medical Research Council (MR/N015665/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Library and Information Sciences › scientometrics and bibliometrics research
- Keywords
- machine learning classifiers, text mining, neuropathic pain, psychosis, vitamin D, multiple sclerosis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.