CoChat

PICO entity extraction for preclinical animal literature.

Open access

The DOI matches its Crossref record; 37 of the 37 references it cites were found in a registry or an index; 4 of the 11 sources its claims cite were checked for support.

Abstract

Background: Natural language processing could assist multiple tasks in systematic reviews to reduce workflow, including the extraction of PICO elements such as study populations, interventions, comparators and outcomes. The PICO framework provides a basis for the retrieval and selection for inclusion of evidence relevant to a specific systematic review question, and automatic approaches to PICO extraction have been developed particularly for reviews of clinical trial findings. Considering the difference between preclinical animal studies and clinical trials, developing separate approaches is necessary. Facilitating preclinical systematic reviews will inform the translation from preclinical to clinical research. Methods: We randomly selected 400 abstracts from the PubMed Central Open Access database which described in vivo animal research and manually annotated these with PICO phrases for Species, Strain, methods of Induction of disease model, Intervention, Comparator and Outcome. We developed a two-stage workflow for preclinical PICO extraction. Firstly we fine-tuned BERT with different pre-trained modules for PICO sentence classification. Then, after removing the text irrelevant to PICO features, we explored LSTM-, CRF- and BERT-based models for PICO entity recognition. We also explored a self-training approach because of the small training corpus. Results: For PICO sentence classification, BERT models using all pre-trained modules achieved an F1 score of over 80%, and models pre-trained on PubMed abstracts achieved the highest F1 of 85%. For PICO entity recognition, fine-tuning BERT pre-trained on PubMed abstracts achieved an overall F1 of 71% and satisfactory F1 for Species (98%), Strain (70%), Intervention (70%) and Outcome (67%). The score of Induction and Comparator is less satisfactory, but F1 of Comparator can be improved to 50% by applying self-training. Conclusions: Our study indicates that of the approaches tested, BERT pre-trained on PubMed abstracts is the best for both PICO sentence classification and PICO entity recognition in the preclinical abstracts. Self-training yields better performance for identifying comparators and strains.

Overview

CoChat Summary

TL;DR

This paper develops a two-stage NLP workflow for extracting PICO elements from preclinical animal abstracts, using BERT sentence classification followed by BERT/LSTM-CRF entity recognition. PubMed-pretrained BERT performed best for both tasks, and self-training improved comparator and strain recognition, though overall entity F1 remained around 71%.

Key findings

  • For PICO sentence classification, all BERT models achieved F1 >80%; PubMedBERT trained on PubMed abstracts had the highest F1 of 85.4% (recall 88.4%, precision 85.0%). Biomedical-domain BERT improved F1 by 4% over general-domain BERT, and pure biomedical pre-training increased recall by 7%.
  • For PICO entity recognition, BERT models outperformed LSTM models by 3–27% F1. The validation-selected best model, PubMedBERT pre-trained on PubMed abstracts and full texts, achieved overall F1 69.9% before self-training and 71.0% after self-training on the gold test set.
  • Entity-level F1 before self-training was Species 98.1%, Intervention 70.2%, Outcome 65.4%, Strain 63.4%, Induction 49.1%, and Comparator 16.0%. Self-training mainly improved Comparator to 48.5% and Strain to 70.0%.
  • Self-training with threshold 0.95 improved best F1 by 5% on validation and 1% on test at the sixth iteration; threshold 0.99 produced no improvement beyond the first iteration.

Methods

  • From 2,207,654 PubMed Central Open Access articles (2010–2019), a citation-screening filter with 99% precision identified 50,653 in vivo animal abstracts. The authors randomly selected 400 abstracts for manual PICO annotation and 10,000 for self-training; annotation yielded 6,837 entities across Species, Strain, Induction, Intervention, Comparator, and Outcome.
  • The workflow split extraction into (1) PICO sentence classification using fine-tuned BERT variants (BERT-base, BioBERT, PubMedBERT-abs, PubMedBERT-full) on 4,247 sentences split 80%/10%/10%, and (2) PICO entity recognition on PICO sentences only, comparing BiLSTM, BiLSTM-CRF, BERT, BERT-CRF, and BERT-BiLSTM-CRF with BIO tagging and entity-level metrics.
  • Self-training used 400 annotated abstracts as gold data and 10,000 unlabelled abstracts as silver data; pseudo-labels were generated by the best entity recognizer, records were selected by average token prediction probability thresholds (0.95 or 0.99), and the enlarged dataset was used to fine-tune a newly initialized recognizer iteratively.

Limitations

  • The paper states that the training corpus is abstract-level, while some PICO elements in preclinical animal studies are often not described in abstracts; this limits usefulness and prevents transfer to full-text identification without further evaluation. It also states that the amount of training, validation, and test data is not adequate, so conclusions may be biased, and that self-training can propagate both knowledge and error from high-confidence predictions.
  • Evident from the text: Comparator and Induction had low F1 scores, and the entity distribution was highly imbalanced (Comparator 1.8%, Strain 3.3%, Outcome 40.6%), likely contributing to poor comparator performance. Annotation ambiguity around PICO sentence definitions and comparator/intervention boundaries is also described in the error analysis.
  • Evident from the text: the entity-recognition test set consisted of only 40 truncated abstracts, so entity-level estimates may be unstable.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1186/s13643-022-02074-4

Cites

37 of 37 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Fig. 1 Preclinical PICO annotation example. Screenshot from tagtog
Fig. 1 Preclinical PICO annotation example. Screenshot from tagtogCC BY · PubMed Central
Fig. 2 The workflow of the PICO extraction
Fig. 2 The workflow of the PICO extractionCC BY · PubMed Central
Fig. 3 The workflow of the self-training in our experiments
Fig. 3 The workflow of the self-training in our experimentsCC BY · PubMed Central
Fig. 4 Performance of PubMedBERT for PICO entity recognition using self-training
Fig. 4 Performance of PubMedBERT for PICO entity recognition using self-trainingCC BY · PubMed Central
Fig. 5 The visualisation of the Streamlit app
Fig. 5 The visualisation of the Streamlit appCC BY · PubMed Central

Verification

Claims the paper makes

4 Valid · 0 Not valid · 7 Not checked · 59 Own results and interpretations

43 more claims

Create a free account to see all 70 claims and to verify the 7 references still open.

See all claims — free account
  • Probably not a substantial claim. 12
  • It may state two separate claims as one. 14

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

2 of 2

Citations

Cites

Show all 37 references

Research works printed but not linked: 4

  1. Higgins JPT, Green S, (editors). Cochrane handbook for systematic reviews of interventions version 5.1.0 [updated March 2011]. 2011.
  2. Loshchilov I, Hutter F. Decoupled weight decay regularization. In: 7th Int Conf Learn Represent ICLR 2019; 2017. http://arxiv.org/abs/1711.05101. Accessed 1 Oct 2020. · arXiv
  3. Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks. In: 30th Int Conf Mach Learn ICML 2013, vol. PART 3; 2012. p. 2347–55. http://arxiv.org/abs/1211.5063. Accessed 18 Nov 2020. · arXiv
  4. Pyysalo S, Ginter F, Moen H, Salakoski T, Ananiadou S. Distributional semantics resources for biomedical text processing. In: Proc 5th Lang Biol Med Conf (LBM 2013); 2013. p. 39–44.

Cited by

See all 31

Metadata

Authors
Wang, Q., Liao, J., Lapata, M., & Macleod, M.
Affiliations
University of Edinburgh (GB)
Corresponding author
Macleod, Malcolm
Venue
Systematic Reviews
Published
2022-09-30
Volume
11
Issue
1
Pages
209-209
ISSN
2046-4053 (ISSN-L)
OpenAlex
W4298110672
Cited by
29
References
29
Citations by year
2026: 6 · 2025: 8 · 2024: 10 · 2023: 4 · 2022: 1
FWCI
1.88
Type
journal article
Language
en
Open access
Licence
Locations
  • Systematic Reviews (publishedVersion, cc-by)
  • Research Square (acceptedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • Edinburgh Research Explorer (University of Edinburgh) (submittedVersion, cc-by)
Funders
  • Chinese Scholarship Council (201809110130)
  • John Climax UK Reproducibility Network PhD Studentship (00001)
Topics
  • Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
  • Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
Keywords
PICO framework, preclinical animal studies, entity recognition, sentence classification, BERT fine-tuning, self-training, pre-trained language models, PubMed abstracts
MeSH terms
Systematic Reviews as Topic, Animals, Language, Natural Language Processing, Publications, PubMed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: PICO entity extraction for preclinical animal literature. https://app.cochat.ai/papers/10.1186/s13643-022-02074-4
The paper
Wang, Q., Liao, J., Lapata, M., & Macleod, M. (2022). PICO entity extraction for preclinical animal literature. Systematic Reviews. https://doi.org/10.1186/s13643-022-02074-4

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.