PICO entity extraction for preclinical animal literature.
Open accessThe DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 4 of the 11 sources its claims cite were checked for support.
Abstract
Background: Natural language processing could assist multiple tasks in systematic reviews to reduce workflow, including the extraction of PICO elements such as study populations, interventions, comparators and outcomes. The PICO framework provides a basis for the retrieval and selection for inclusion of evidence relevant to a specific systematic review question, and automatic approaches to PICO extraction have been developed particularly for reviews of clinical trial findings. Considering the difference between preclinical animal studies and clinical trials, developing separate approaches is necessary. Facilitating preclinical systematic reviews will inform the translation from preclinical to clinical research. Methods: We randomly selected 400 abstracts from the PubMed Central Open Access database which described in vivo animal research and manually annotated these with PICO phrases for Species, Strain, methods of Induction of disease model, Intervention, Comparator and Outcome. We developed a two-stage workflow for preclinical PICO extraction. Firstly we fine-tuned BERT with different pre-trained modules for PICO sentence classification. Then, after removing the text irrelevant to PICO features, we explored LSTM-, CRF- and BERT-based models for PICO entity recognition. We also explored a self-training approach because of the small training corpus. Results: For PICO sentence classification, BERT models using all pre-trained modules achieved an F1 score of over 80%, and models pre-trained on PubMed abstracts achieved the highest F1 of 85%. For PICO entity recognition, fine-tuning BERT pre-trained on PubMed abstracts achieved an overall F1 of 71% and satisfactory F1 for Species (98%), Strain (70%), Intervention (70%) and Outcome (67%). The score of Induction and Comparator is less satisfactory, but F1 of Comparator can be improved to 50% by applying self-training. Conclusions: Our study indicates that of the approaches tested, BERT pre-trained on PubMed abstracts is the best for both PICO sentence classification and PICO entity recognition in the preclinical abstracts. Self-training yields better performance for identifying comparators and strains.
Overview
CoChat Summary
TL;DR
This paper develops a two-stage NLP workflow for extracting PICO elements from preclinical animal abstracts, using BERT sentence classification followed by BERT/LSTM-CRF entity recognition. PubMed-pretrained BERT performed best for both tasks, and self-training improved comparator and strain recognition, though overall entity F1 remained around 71%.
Key findings
- For PICO sentence classification, all BERT models achieved F1 >80%; PubMedBERT trained on PubMed abstracts had the highest F1 of 85.4% (recall 88.4%, precision 85.0%). Biomedical-domain BERT improved F1 by 4% over general-domain BERT, and pure biomedical pre-training increased recall by 7%.
- For PICO entity recognition, BERT models outperformed LSTM models by 3–27% F1. The validation-selected best model, PubMedBERT pre-trained on PubMed abstracts and full texts, achieved overall F1 69.9% before self-training and 71.0% after self-training on the gold test set.
- Entity-level F1 before self-training was Species 98.1%, Intervention 70.2%, Outcome 65.4%, Strain 63.4%, Induction 49.1%, and Comparator 16.0%. Self-training mainly improved Comparator to 48.5% and Strain to 70.0%.
- Self-training with threshold 0.95 improved best F1 by 5% on validation and 1% on test at the sixth iteration; threshold 0.99 produced no improvement beyond the first iteration.
Methods
- From 2,207,654 PubMed Central Open Access articles (2010–2019), a citation-screening filter with 99% precision identified 50,653 in vivo animal abstracts. The authors randomly selected 400 abstracts for manual PICO annotation and 10,000 for self-training; annotation yielded 6,837 entities across Species, Strain, Induction, Intervention, Comparator, and Outcome.
- The workflow split extraction into (1) PICO sentence classification using fine-tuned BERT variants (BERT-base, BioBERT, PubMedBERT-abs, PubMedBERT-full) on 4,247 sentences split 80%/10%/10%, and (2) PICO entity recognition on PICO sentences only, comparing BiLSTM, BiLSTM-CRF, BERT, BERT-CRF, and BERT-BiLSTM-CRF with BIO tagging and entity-level metrics.
- Self-training used 400 annotated abstracts as gold data and 10,000 unlabelled abstracts as silver data; pseudo-labels were generated by the best entity recognizer, records were selected by average token prediction probability thresholds (0.95 or 0.99), and the enlarged dataset was used to fine-tune a newly initialized recognizer iteratively.
Limitations
- The paper states that the training corpus is abstract-level, while some PICO elements in preclinical animal studies are often not described in abstracts; this limits usefulness and prevents transfer to full-text identification without further evaluation. It also states that the amount of training, validation, and test data is not adequate, so conclusions may be biased, and that self-training can propagate both knowledge and error from high-confidence predictions.
- Evident from the text: Comparator and Induction had low F1 scores, and the entity distribution was highly imbalanced (Comparator 1.8%, Strain 3.3%, Outcome 40.6%), likely contributing to poor comparator performance. Annotation ambiguity around PICO sentence definitions and comparator/intervention boundaries is also described in the error analysis.
- Evident from the text: the entity-recognition test set consisted of only 40 truncated abstracts, so entity-level estimates may be unstable.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures





Verification
Claims the paper makes
4 Valid · 0 Not valid · 7 Not checked · 59 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Abstracts that are pre-structured according to the PICO frame or combined demonstration with PICO phrases enable faster judgement of study relevance for each PICO element [4].”
§ PICO entity extraction for preclinical animal literature > Background
What the sources say
- Partly supportedcites [4] Jin D, Szolovits P. PICO element detection in medical text via long short-term memory neural networks. In: Proceedings of the BioNLP 2018 workshop. Stroudsburg: Association for Computational Linguistics; 2018. p. 67–75. https://doi.org/10.18653/v1/W18-2308. sourceExperimental — accuracy not yet measured
Our copy supports only the general idea, and with weaker warrant than the claim carries: the Introduction (p. 1) says it 'would be advantageous that all papers (or at least their abstracts) can be organized according to the PICO foci' to facilitate the selection of potentially relevant papers — a hedged design rationale, not a demonstrated speed-up; and the Related Work (p. 2) reports that clinicians' use of the PICO framework 'has been validated for its performance improvement on searching literature for clinical questions', explicitly crediting prior user studies (Schardt et al., 2007; Boudin et al., 2010c; Znaidi et al., 2015) rather than showing it itself. Nothing in our copy states faster judgement of study relevance 'for each PICO element', nor the 'combined with PICO phrases' variant; the paper's own contribution is automatic LSTM-based PICO sentence labelling and a curated PubMed dataset. So the source backs part of the attributed point (PICO structuring aids literature searching/selection) but not the specific speed/relevance-per-element assertion, and it is a reporter rather than the originator of the user-study finding. (LLM, conf=0.72)
boostedstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
Every source the paper cites for this claim was checked and supports it.
In the paper
“For example, in animal studies, disease is not naturally present but often induced, different species can be used, and outcomes of interest can include survival, behavioural, histological and biochemical outcomes [7].”
§ PICO entity extraction for preclinical animal literature > Background
What the sources say
- Supportedcites [7] Hooijmans CR, Rovers MM, De Vries RBM, Leenaars M, Ritskes-Hoitinga M, Langendam MW. SYRCLE’s risk of bias tool for animal studies. BMC Med Res Methodol. 2014;14:43. https://doi.org/10.1186/1471-2288-14-43. source
Table 1 in the Background (p. 1) contrasts RCTs ('Disease naturally present') with animal intervention studies ('Disease often induced (with unclear/insufficient similarity to the human condition)'), which is exactly what the citing sentence attributes to Hooijmans et al. The Results section reinforces this, e.g. the discussion of 'Timing of disease induction' in the randomization/blinding entries. (LLM, conf=0.93)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Biomedical-domain BERT improves F1 score by 4% compared with general-domain BERT, and BERT with pure biomedical-domain pre-training (two PuBMedBERT) can identify more PICO sentences than BERT with general pre-training (BERT-base) or mixed-domain pre-training (BioBERT), as recall increased by 7%.”
§ PICO entity extraction for preclinical animal literature > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The score of Induction and Comparator is less satisfactory, but F1 of Comparator can be improved to 50% by applying self-training.”
§ PICO entity extraction for preclinical animal literature > Abstract > Results
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“F1 scores of Strain and Induction are 63% and 49%, respectively, so there remains room for improvement.”
§ PICO entity extraction for preclinical animal literature > Results
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
43 more claims
Create a free account to see all 70 claims and to verify the 7 references still open.
- Probably not a substantial claim. 12
- It may state two separate claims as one. 14
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“400 RCT abstracts in firstcorpus,10k abstract in additional corpus from mined data”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [82]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ PICO entity extraction for preclinical animal literature > Abstract > Results
“fine-tuning BERT pre-trained on PubMed abstracts achieved an overall F1 of 71%”
§ PICO entity extraction for preclinical animal literature > Conclusions
“BERT pre-trained on PubMed abstracts and full texts is optimal for PICO entity recognition tasks in preclinical abstracts.”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“In total, 6837 entities were annotated across 400 abstracts”
§ PICO entity extraction for preclinical animal literature > Methods > Dataset
Reported 17.5 · recomputed 17.0925
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
Advances in Neural Information Processing Systems 29
Teymur, O., Zygalakis, K., & Calderhead, B. (2016) · Cited by 13,887
DeepWalk: online learning of social representations
Perozzi, B., Al-Rfou, R., & Skiena, S. (2014) · Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2623330.2623732 · Cited by 10,123
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
SYRCLE’s risk of bias tool for animal studies
Hooijmans, C. R., Rovers, M. M., de Vries, R. B., Leenaars, M., Ritskes-Hoitinga, M., & Langendam, M. W. (2014) · BMC Medical Research Methodology · doi:10.1186/1471-2288-14-43 · Cited by 4,415 · cited 1×
“For example, in animal studies, disease is not naturally present but often induced, different species can be used, and outcomes of interest can include survival, behavioural, histological and biochemical outcomes [7].”
Background
Research works printed but not linked: 4
- Higgins JPT, Green S, (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. 2011.
- Loshchilov I, Hutter F. Decoupled weight decay regularization. In: 7th Int Conf Learn Represent ICLR 2019; 2017. http://arxiv.org/abs/1711.05101. Accessed 1 Oct 2020. · arXiv
- Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks. In: 30th Int Conf Mach Learn ICML 2013, vol. PART 3; 2012. p. 2347–55. http://arxiv.org/abs/1211.5063. Accessed 18 Nov 2020. · arXiv
- Pyysalo S, Ginter F, Moen H, Salakoski T, Ananiadou S. Distributional semantics resources for biomedical text processing. In: Proc 5th Lang Biol Med Conf (LBM 2013); 2013. p. 39–44.
Cited by
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2024) · arXiv · doi:10.48550/arxiv.2409.04600 · Cited by 156
Scherbakov, D., Hubig, N., Jansari, V., Bakumenko, A., & Lenert, L. A. (2025) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaf063 · Cited by 154
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Tóth, B., Berek, L., Gulàcsi, L., Péntek, M., & Zrubka, Z. (2024) · Systematic Reviews · doi:10.1186/s13643-024-02592-3 · Cited by 71
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 74
Metadata
- Authors
- Wang, Q., Liao, J., Lapata, M., & Macleod, M.
- Affiliations
- University of Edinburgh (GB)
- Corresponding author
- Macleod, Malcolm
- Venue
- Systematic Reviews
- Published
- 2022-09-30
- Volume
- 11
- Issue
- 1
- Pages
- 209-209
- ISSN
- 2046-4053 (ISSN-L)
- PMC
- PMC9524079
- OpenAlex
- W4298110672
- Semantic Scholar
- 8816f3e61fc0e1e467da7922503750ecfb420283
- Cited by
- 29
- References
- 29
- Citations by year
- 2026: 6 · 2025: 8 · 2024: 10 · 2023: 4 · 2022: 1
- FWCI
- 1.88
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- cc-by
- creativecommons.org/licenses/by/4.0 (tdm)
- creativecommons.org/licenses/by/4.0 (vor)
- Locations
- Systematic Reviews (publishedVersion, cc-by)
- Research Square (acceptedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
- Funders
- Chinese Scholarship Council (201809110130)
- John Climax UK Reproducibility Network PhD Studentship (00001)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Keywords
- PICO framework, preclinical animal studies, entity recognition, sentence classification, BERT fine-tuning, self-training, pre-trained language models, PubMed abstracts
- MeSH terms
- Systematic Reviews as Topic, Animals, Language, Natural Language Processing, Publications, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: PICO entity extraction for preclinical animal literature. https://app.cochat.ai/papers/10.1186/s13643-022-02074-4
- The paper
- Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022). PICO entity extraction for preclinical animal literature. Systematic Reviews. https://doi.org/10.1186/s13643-022-02074-4
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.