Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Open accessThe DOI matches its Crossref record; 24 of the 24 references it cites were found in a registry or an index.
Abstract
PICO (Population/problem, Intervention, Comparison, and Outcome) is widely adopted for formulating clinical questions to retrieve evidence from the literature. It plays a crucial role in Evidence-Based Medicine (EBM). This paper contributes a scalable deep learning method to extract PICO statements from RCT articles. It was trained on a small set of richly annotated PubMed abstracts using an LSTM-CRF model. By initializing our model with pretrained parameters from a large related corpus, we improved the model performance significantly with a minimal feature set. Our method has advantages in minimizing the need for laborious feature handcrafting and in avoiding the need for large shared annotated data by reusing related corpora in pretraining with a deep neural network.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Figures




Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“described that the same problem needs to be addressed through stratification of the positive instances of each class within folds.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“In terms of system evaluation, 10-fold cross-validation causes high variance in results when using small datasets such as NICTA-PIBOSO, 54 , 85 , 104”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
Experimental — accuracy not yet measuredIn the citing paper
“Two publications 55,85 cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [85]
In the citing paper
“Similar disagreements, 65,85,104 along with missed annotations, 72 are time-intensive to reconciliate 97 and make the scores less reliable.”
From Data extraction methods for systematic review (semi)automation: Update of a living systematic review (2025) · cites this paper as [85]
In the citing paper
“Two publications 55 , 85 cautioned that cross-validation with a high amount of folds (e.g. 10) causes high variance in evaluation results when using small datasets such as NICTA-PIBOSO.”
From Data extraction methods for systematic review (semi)automation: a living systematic review (2021) · cites this paper as [85]
2 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Efficient Estimation of Word Representations in Vector Space
Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013) · arXiv · doi:10.48550/arxiv.1301.3781 · Cited by 26,568 · cited 1×
“While the light green, namely word embedding is pretrained on an entire collection of abstracts on PubMed from 1990–2018 using word2vec toolkit [21].”
MethodBioBERT: a pre-trained biomedical language representation model for biomedical text mining
Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., & Kang, J. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btz682 · Cited by 7,218
Neural Architectures for Named Entity Recognition
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/n16-1030 · Cited by 4,401
Utilization of the PICO framework to improve searching PubMed for clinical questions
Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-16 · Cited by 3,053
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · cited 2×
“The PICO framework is specialized to help break down the need for evidence into searchable keywords and to formulate answerable research questions [4].”
BackgroundEnd-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
Ma, X., & Hovy, E. (2016) · Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p16-1101 · Cited by 2,567 · cited 1×
“The base model is similar to [14], and is also used in EBM-NLP corpus paper to generate task baseline for identifying PICO span.”
MethodSeventy-five trials and eleven systematic reviews a day: how will we ever keep up?
Bastian, H., Glasziou, P., & Chalmers, I. (2010) · PLoS Medicine · doi:10.1371/journal.pmed.1000326 · Cited by 1,188 · cited 1×
“It is practically impossible to catch up with the explosion of the biomedical scientific literature and realize fast and effective evidence retrieval and decision making for EBM practitioners [2; 3].”
BackgroundNeural Architectures for Named Entity RecognitionPreprint
Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., & Dyer, C. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1603.01360 · Cited by 766
Evidence Based Medicine - New Approaches and Challenges
Masic, I., Miokovic, M., & Muhamedagic, B. (2008) · Acta Informatica Medica · doi:10.5455/aim.2008.16.219-225 · Cited by 495 · cited 1×
“Evidence-Based Medicine (EBM) is the conscientious, explicit, judicious and reasonable use of modern, best evidence in making decisions about the care of individual patients [1].”
BackgroundAnswering Clinical Questions with Knowledge-Based and Statistical Techniques
Demner-Fushman, D., & Lin, J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.1.63 · Cited by 296 · cited 1×
“During the last decade, the primary solutions have evolved from knowledge-based to statistical-based such as Support Vector Machine (SVM) and Conditional Random Field (CRF) [8–11].”
Background
Research works printed but not linked: 1
- [22]22. Soldaini L and Goharian N, Quickumls: a fast, unsupervised approach for medical concept extraction, in: MedIR workshop, sigir, 2016.
Cited by
Pandey, B., Pandey, D. K., Mishra, B. P., & Rhmann, W. (2021) · Journal of King Saud University - Computer and Information Sciences · doi:10.1016/j.jksuci.2021.01.007 · Cited by 157
Data extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Santos, Á. O. D., da Silva, E. S., Couto, L. M., Reis, G. V. L., & Belo, V. S. (2023) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2023.104389 · Cited by 64
UMLS-based data augmentation for natural language processing of clinical research literature
Kang, T., Perotte, A., Tang, Y., Ta, C., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa309 · Cited by 55
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
Yeun, Y.-R., & Kim, S.-D. (2022) · International Journal of Environmental Research and Public Health · doi:10.3390/ijerph19031624 · Cited by 41
“The inclusion criteria were evaluated using the PICO (population, intervention, comparison, and outcome) elements in RCTs [19].”
BackgroundMethodData extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
Effects of yoga on immune function: A systematic review of randomized controlled trials
Yeun, Y.-R., & Kim, S.-D. (2021) · Complementary Therapies in Clinical Practice · doi:10.1016/j.ctcp.2021.101446 · Cited by 24
Nagel, J., Wegener, F., Grim, C., & Hoppe, M. W. (2023) · JMIR mhealth and uhealth · doi:10.2196/50616 · Cited by 23
“Eligibility criteria according to the population, intervention, comparison, outcome, study design (PICOS) scheme [16] were applied.”
MethodHu, Y., Keloth, V. K., Raja, K., Chen, Y., & Xu, H. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad542 · Cited by 21
“A recent study showed improved performance on PICO extraction when the NER model was first pre-trained on the EBM-NLP corpus, and further fine-tuned with the additional data annotated by themselves (Kang et al. 2019).”
MethodWhitton, J., & Hunter, A. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102661 · Cited by 20
Zhang, G., Zhou, Y., Hu, Y., Xu, H., Weng, C., & Peng, Y. (2024) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocae065 · Cited by 17
Yang, Y., Cao, Z., Zhao, P., Zeng, D. D., Zhang, Q., & Luo, Y. (2021) · Journal of Safety Science and Resilience · doi:10.1016/j.jnlssr.2021.08.002 · Cited by 16
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad036 · Cited by 14
Self-citationGosak, L., Štiglic, G., Pruinelli, L., & Vrbnjak, D. (2024) · Journal of Nursing Scholarship · doi:10.1111/jnu.13036 · Cited by 21
A neuro-symbolic method for understanding free-text medical evidence.
Kang, T., Turfah, A., Kim, J., Perotte, A., & Weng, C. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab077 · Cited by 11
Self-citationBioDEX: Large-Scale Biomedical Adverse Drug Event Extraction for Real-World Pharmacovigilance
D’Oosterlinck, K., Remy, F., Deleu, J., Demeester, T., Develder, C., Zaporojets, K., Ghodsi, A., Ellershaw, S., Collins, J., & Potts, C. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.896 · Cited by 24
Lan, M., Cheng, M., Hoang, L., ter Riet, G., & Kilicoglu, H. (2024) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2024.104628 · Cited by 15
McKivigan, J. M. (2020) · Higher Education Research · doi:10.11648/j.her.20200502.14 · Cited by 8
“, the PICO system can be applied to reduce and divide evidence into keywords that can be easily researched [18].”
MethodWitte, C., Schmidt, D. M., & Cimiano, P. (2024) · Journal of Biomedical Semantics · doi:10.1186/s13326-024-00305-2 · Cited by 7
“Other methods work with Conditional Random Fields (CRFs) in combination with (Bi-)LSTMs (e.g., Jin and Szolovits [46], Kang et al. [47]) or rule-based methods (e.g., Chabou and Iglewski [48]).”
MethodEfficient evidence selection for systematic reviews in traditional Chinese medicine
Li, Y., Huang, Z., Luan, Z., Xu, S., Zhang, Y., Wu, L., Wu, D., Han, D., & Liu, Y. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-024-02430-z · Cited by 9
The role of breathing techniques in the management of asthma: a systematic review
Zafar, S. N., Khalid, M., Jamil, A., Butt, S. K., Manzoor, A., & Nadeem, M. (2024) · Journal of the Pakistan Medical Association · doi:10.47391/jpma.8595 · Cited by 7
Newton, A. J. H., Chartash, D., Kleinstein, S. H., & McDougal, R. A. (2023) · BMC Bioinformatics · doi:10.1186/s12859-023-05397-8 · Cited by 5
“the metadata suggestions for ModelDB [9], and PICO (population, intervention, control, and outcomes) extraction [10], which tags words related to the PICO elements in randomized control trials.”
MethodA Sample Size Extractor for RCT Reports
Lin, F., Liu, H., Moon, P., & Weng, C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220151 · Cited by 6
Paraskevopoulos, S., Smeets, P., Tian, X., & Medema, G. J. (2022) · International Journal of Hygiene and Environmental Health · doi:10.1016/j.ijheh.2022.114018 · Cited by 4
Ji, B., Batubara, I. M. S., Batten, J., Peng, X., Chen, S., & Ni, Z. (2025) · JMIR Research Protocols · doi:10.2196/68677 · Cited by 4
“Comparators or comparisons refer to groups that were compared to the intervention group [33].”
A Scoping Review of Adopted Information Extraction Methods for RCTs.
Aletaha, A., Nemati-Anaraki, L., Keshtkar, A., Sedghi, S., Keramatfar, A., & Korolyova, A. (2023) · Medical Journal of The Islamic Republic of Iran · doi:10.47176/mjiri.37.95 · Cited by 2
“The field of NLP is growing rapidly to become one of the most active research areas in trial studies that extract key features, including different PICO elements and especially outcomes, which is the basis for determining the sample size for RCT design (43).”
BackgroundPeeters, C., Vijverberg, K., Pouwer, M. G., Westerman, B. A., Boot, M., & Verberne, S. (2025) · BMC Medical Research Methodology · doi:10.1186/s12874-025-02624-z · Cited by 6
“The EBM-NLP corpus was used to train PICO-extracting systems on a sentence [32, 33] and span level [34, 35].”
Monteiro, J. P., Morine, M. J., Ued, F. D. V., & Kaput, J. (2023) · Nutrients · doi:10.3390/nu15020270 · Cited by 2
“, decreased biomarker or disease symptom)—PICO [53]—for each study in the corpus, which was beyond the scope of the topic for this review.”
BackgroundPre-Training to Identify Immunization-Related Entities from Systematic Reviews
İlgen, B., Pilic, A., Harder, T., & Hattab, G. (2023) · International Conference on Natural Language Processing and Information Retrieval (NLPIR) · doi:10.1145/3639233.3639355 · Cited by 4
“Formulating the relevant questions is an initial step in the IE domain [8].”
BackgroundMethodRachmayanti, R. D., Dewi, F. S. T., Setiyawati, D., Megatsari, H., Diana, R., & Vinarti, R. A. (2024) · JMIR Research Protocols · doi:10.2196/58681 · Cited by 8
“Outcome refers to the measure of interest or the study’s target end points that are objectively defined.”
Extending PICO with Observation Normalization for Evidence Computing
Turfah, A., Liu, H., Stewart, L. A., Kang, T., & Weng, C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220076 · Cited by 5
Rachmayanti, R. D., Dewi, F. S. T., Setiyawati, D., Megatsari, H., Diana, R., & Vinarti, R. A. (2024) · Research Square · doi:10.21203/rs.3.rs-3440896/v1 · Cited by 2
Kaur, R., & Singh, S. (2026) · IISE Transactions on Healthcare Systems Engineering · doi:10.1080/24725579.2026.2619123 · Cited by 1
Buddrus, U., Kutza, J.-O., Thye, J., Esdar, M., Hübner, U. H., & Liebe, J.-D. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1755598 · Cited by 7
“The main reasons are imprecisely formulated research questions, ineffective search and screening strategies, and fragmented databases (5).”
Representation and Normalization of Complex Interventions for Evidence Computing
Chen, Z., Liu, H., Liao, S., Bernard, M., Kang, T., Stewart, L. A., & Weng, C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220146 · Cited by 1
Reproducibility in Named Entity Recognition: A Case Study Analysis
Villarmin, C. C., Cohen‐Boulakia, S., & Naderi, N. (2024) · IEEE International Conference on eScience · doi:10.1109/e-science62913.2024.10678721 · Cited by 3
“For example, in NER, metrics can be computed using two different approaches, as it has been previously discussed and reported in [13], [25].”
Data-Driven Modeling of Randomized Controlled Trial Outcomes
Chen, Z., Fang, Y., Liu, H., & Weng, C. (2022) · Studies in health technology and informatics · doi:10.3233/shti220481 · Cited by 2
Peeters, C., Vijverberg, K., Pouwer, M. G., Westerman, B. A., Boot, M., & Verberne, S. (2024) · medRxiv · doi:10.1101/2024.05.31.24308278 · Cited by 1
Pipeline for retrieval of COVID-19 immune signaturesPreprint
Newton, A. J. H., Chartash, D., Kleinstein, S. H., & McDougal, R. A. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.12.29.474353
“the metadata suggestions for ModelDB [9], and PICO (population, intervention, control, and outcomes) extraction [10], which tags words related to the PICO elements in randomized control trials.”
MethodMotahari-Nezhad, H. (2023) · CUB Ph.D. Dissertations (Corvinus University of Budapest) · doi:10.14267/phd.2023046 · Cited by 3
Literature Filtering for Systematic Reviews with Transformers
Hawkins, J., & Tivey, D. R. (2024) · International Conference on Communications, Computing and Artificial Intelligence (CCCAI) · doi:10.1145/3676581.3676582 · Cited by 3
Buddrus, U., & Liebe, J.-D. (2025) · doi:10.2196/preprints.78210
Buddrus, U., & Liebe, J.-D. (2026) · Online Journal of Public Health Informatics · doi:10.2196/78210
“The PICO (population, problem, or patient, intervention, comparison, and outcome) framework has been widely adopted in clinical research to formulate answerable questions and design efficient search strategies [3-6].”
Huang, Y., Wang, L., Yang, X., & Tang, X. (2026) · doi:10.2196/preprints.93396
Metadata
- Authors
- Tian, K., Shirui, Z., & Chunhua, W.
- Affiliations
- Columbia University (US)
- Venue
- Studies in Health Technology and Informatics
- Publisher
- IOS Press
- Published
- 2019
- Volume
- 264
- Pages
- 188-192
- ISSN
- 0926-9630 (ISSN-L)
- PMC
- PMC6852618
- OpenAlex
- W2980993994
- Cited by
- 45
- References
- 21
- Citations by year
- 2026: 4 · 2025: 5 · 2024: 10 · 2023: 12 · 2022: 6 · 2021: 6 · 2020: 2
- FWCI
- 3.85
- Type
- book chapter
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by-nc
- Locations
- Studies in health technology and informatics (publishedVersion, cc-by-nc)
- PubMed (publishedVersion)
- PubMed Central (acceptedVersion, cc-by-nc)
- PubMed
- Funders
- National Institutes of Health (OT3TR002027, UL1TR001873)
- U.S. National Library of Medicine (R01LM009886)
- National Center for Advancing Translational Sciences
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Keywords
- randomized controlled trials, clinical question formulation, evidence-based medicine, information extraction, LSTM-CRF, pre-trained language models, PubMed abstracts
- MeSH terms
- Randomized Controlled Trials as Topic, Neural Networks, Computer, Evidence-Based Medicine, PubMed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.