Improving reference prioritisation with PICO recognition.
Open accessThe DOI matches its Crossref record; 136 of the 136 references it cites were found in a registry or an index.
Abstract
Background: Machine learning can assist with multiple tasks during systematic reviews to facilitate the rapid retrieval of relevant references during screening and to identify and extract information relevant to the study characteristics, which include the PICO elements of patient/population, intervention, comparator, and outcomes. The latter requires techniques for identifying and categorising fragments of text, known as named entity recognition. Methods: A publicly available corpus of PICO annotations on biomedical abstracts is used to train a named entity recognition model, which is implemented as a recurrent neural network. This model is then applied to a separate collection of abstracts for references from systematic reviews within biomedical and health domains. The occurrences of words tagged in the context of specific PICO contexts are used as additional features for a relevancy classification model. Simulations of the machine learning-assisted screening are used to evaluate the work saved by the relevancy model with and without the PICO features. Chi-squared and statistical significance of positive predicted values are used to identify words that are more indicative of relevancy within PICO contexts. Results: Inclusion of PICO features improves the performance metric on 15 of the 20 collections, with substantial gains on certain systematic reviews. Examples of words whose PICO context are more precise can explain this increase. Conclusions: Words within PICO tagged segments in abstracts are predictive features for determining inclusion. Combining PICO annotation model into the relevancy classification pipeline is a promising approach. The annotations may be useful on their own to aid users in pinpointing necessary information for data extraction, or to facilitate semantic search.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures
![Fig. 1 PICO recognition example. Visualisation of the trained model’s predictions of PICO elements within a reference (title and abstract) from the Proton Pump Inhibitors review. The interventions tags correspond to drug names, participant spans cover characteristics of the population, but erroneously include details of the intervention. The latter demonstrates the model’s ability to nest shorter spans within longer pans. The outcomes cover spans for qualitative and quantitative measures. Screenshot from the brat system [23]](https://figures.cochat.ai/paper-figures/c3cee29f68be446b8e348de18ef05211/improving-reference-prioritisation-with-pico-recognition-figure-1.jpg)


Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. 59,96,106”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [106]
The cited source says what the claim attributes to it.
In the citing paper
“Publications in the neural and deep-learning domain described approaches such as early stopping, dropout, L2-regularisation, or weight decay. 59 , 96 , 106”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [106]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Glove: Global Vectors for Word Representation
Pennington, J., Socher, R., & Manning, C. (2014) · Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.3115/v1/d14-1162 · Cited by 34,071
(2014) · Cited by 33,419
(2003) · Journal of Machine Learning Research · doi:10.5555/944919.944937 · Cited by 32,337
Distributed Representations of Words and Phrases and their Compositionality
Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1310.4546 · Cited by 24,533
Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
Lafferty, J., McCallum, A., & Pereira, F. C. N. (2001) · ScholarlyCommons (University of Pennsylvania) · Cited by 12,935
Advances in Neural Information Processing Systems 25
Zhang, Y., Sutton, C. A., Storkey, A., & Zoubin Ghahramani, A. (2012) · Cited by 12,653
Research works printed but not linked: 6
- Kanoulas E, Li D, Azzopardi L, Spijker R. Clef 2018 technologically assisted reviews in empirical medicine overview. In: CEUR Workshop Proceedings, vol. 2125: 2018. p. 1–34. http://ceur-ws.org/Vol-2125/invited_paper_6.pdf. Accessed 27 Sept 2018.
- Demner-Fushman D, Lin J. Knowledge extraction for clinical question answering: Preliminary results. In: Proceedings of the AAAI-05 Workshop on Question Answering in Restricted Domains. Pittsburgh: AAAI Press (American Association for Artificial Intelligence): 2005. p. 9–13.
- Huang Z, Xu W, Yu K. Bidirectional LSTM-CRF models for sequence tagging. arXiv preprint arXiv:1508.01991. 2015. · arXiv
- Sætre R, Yoshida K, Yakushiji A, Miyao Y, Matsubayashi Y, Ohta T. AKANE system: protein-protein interaction pairs in BioCreAtIvE2 challenge, PPI-IPS subtask. In: Proceedings of the Second Biocreative Challenge Workshop, vol. 209. Madrid: CNIO Centro Nacional de Investigaciones Oncológicas: 2007. p. 212.
- Bengio Y, Ducharme R, Vincent P, Jauvin C. A neural probabilistic language model. J Mach Learn Res. 2003; 3(Feb):1137–55.
- Collaborative Approach to Meta-Analysis and Review of Animal Data from Experimental Studies. CAMARADES. www.camarades.info. http://www.dcn.ed.ac.uk/camarades/default.htm.
Cited by
Frandsen, T. F., Bruun Nielsen, M. F., Lindhardt, C. L., & Eriksen, M. B. (2020) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2020.07.005 · Cited by 251
Barriers and facilitators of HPV vaccination in sub-saharan Africa: a systematic review
Kutz, J.-M., Rausche, P., Gheit, T., Puradiredja, D. I., & Fusco, D. (2023) · BMC Public Health · doi:10.1186/s12889-023-15842-1 · Cited by 204
Pre-trained language models with domain knowledge for biomedical extractive summarization
Xie, Q., Bishop, J. A., Tiwari, P., & Ananiadou, S. (2022) · Knowledge-Based Systems · doi:10.1016/j.knosys.2022.109460 · Cited by 120
Self-citationData extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 78
Metadata
- Authors
- Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S.
- Affiliations
- University of Manchester (GB)
- Corresponding author
- Ananiadou, Sophia
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2019-12
- Volume
- 19
- Issue
- 1
- Pages
- 256-256
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC6896258
- OpenAlex
- W2992824360
- Cited by
- 62
- References
- 134
- Citations by year
- 2026: 8 · 2025: 9 · 2024: 14 · 2023: 12 · 2022: 8 · 2021: 7 · 2020: 4
- FWCI
- 3.18
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor, 4 d)
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Research Explorer (The University of Manchester) (publishedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Funders
- University of Manchester
- Narodowa Agencja Wymiany Akademickiej (PPN/PPO/2018/1/00006)
- Directorate for Biological Sciences
- Medical Research Council (MR/L01078X/1, MR/N00583X/1)
- Biotechnology and Biological Sciences Research Council (BB/P025684/1)
- Topics
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- PICO, named entity recognition, systematic reviews, bibliographic abstracts, recurrent neural networks, biomedical text annotation, data extraction, semantic search
- MeSH terms
- Machine Learning, Systematic Reviews as Topic, Humans, Semantics, Databases, Genetic, Information Dissemination
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Improving reference prioritisation with PICO recognition. https://app.cochat.ai/papers/10.1186/s12911-019-0992-8
- The paper
- Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019). Improving reference prioritisation with PICO recognition. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-019-0992-8
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.