The DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Open accessAbstract
PICO (Population/problem, Intervention, Comparison, and Outcome) is widely adopted for formulating clinical questions to retrieve evidence from the literature. It plays a crucial role in Evidence-Based Medicine (EBM). This paper contributes a scalable deep learning method to extract PICO statements from RCT articles. It was trained on a small set of richly annotated PubMed abstracts using an LSTM-CRF model. By initializing our model with pretrained parameters from a large related corpus, we improved the model performance significantly with a minimal feature set. Our method has advantages in minimizing the need for laborious feature handcrafting and in avoiding the need for large shared annotated data by reusing related corpora in pretraining with a deep neural network.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“described that the same problem needs to be addressed through stratification of the positive instances of each class within folds.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In terms of system evaluation, 10-fold cross-validation causes high variance in results when using small datasets such as NICTA-PIBOSO, 54 , 85 , 104”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
Experimental — accuracy not yet measuredIn the citing paper
“Two publications 55,85 cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [85]
In the citing paper
“Similar disagreements, 65,85,104 along with missed annotations, 72 are time-intensive to reconciliate 97 and make the scores less reliable.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [85]
In the citing paper
“Two publications 55 , 85 cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568
BioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
Utilization of the PICO framework to improve searching PubMed for clinical questions
Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-16 · Cited by 3,053
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · cited 2×
“The PICO framework is specialized to help break down the need for evidence into searchable keywords and to formulate answerable research questions [4].”
BackgroundEnd-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567
Seventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188
Neural Architectures for Named Entity RecognitionPreprint
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01360 · Cited by 766
Evidence Based Medicine - New Approaches and Challenges
Masic, I., Miokovic, M., & Muhamedagic, B. (2008) · Acta Informatica Medica · doi:10.5455/aim.2008.16.219-225 · Cited by 495
Answering Clinical Questions with Knowledge-Based and Statistical Techniques
Demner-Fushman, D., & Lin, J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.1.63 · Cited by 296
Research works printed but not linked: 1
- [22]22. Soldaini L and Goharian N, Quickumls: a fast, unsupervised approach for medical concept extraction, in: MedIR workshop, sigir, 2016.
Cited by
Pandey, B., Pandey, D. K., Mishra, B. P., & Rhmann, W. (2021) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.01.007 · Cited by 157
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
dos Santos, Á. O., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 78
UMLS-based data augmentation for natural language processing of clinical research literature
Kang, T., Perotte, A., Tang, Y., Ta, C., & Weng, C. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa309 · Cited by 58
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 74
Metadata
- Authors
- Tian, K., Shirui, Z., & Chunhua, W.
- Affiliations
- Columbia University (US)
- Venue
- Studies in Health Technology and Informatics
- Publisher
- IOS Press
- Published
- 2019
- Volume
- 264
- Pages
- 188-192
- ISSN
- 0926-9630 (ISSN-L)
- PMC
- PMC6852618
- OpenAlex
- W2980993994
- Cited by
- 45
- References
- 21
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 10 · 2023: 12 · 2022: 6 · 2021: 6 · 2020: 2
- FWCI
- 3.85
- Type
- book chapter
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Studies in health technology and informatics (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc)
- PubMed
- Funders
- National Institutes of Health (OT3TR002027, UL1TR001873)
- U.S. National Library of Medicine (R01LM009886)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- randomized controlled trials, clinical question formulation, evidence-based medicine, information extraction, LSTM-CRF, pre-trained language models, PubMed abstracts
- MeSH terms
- Randomized Controlled Trials as Topic, Neural Networks, Computer, Evidence-Based Medicine, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature. https://app.cochat.ai/papers/10.3233/shti190209
- The paper
- Tian, K., Shirui, Z., & Chunhua, W. (2019). Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature. Studies in Health Technology and Informatics. https://doi.org/10.3233/shti190209
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.