Classification of PICO elements by text features systematically extracted from PubMed abstracts.
The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Abstract
We propose and evaluate a systematic approach to detect and classify Patient/Problem, Intervention, Comparison and Outcome (PICO) from the medical literature. The training and test corpora were generated systematically and automatically from structured PubMed abstracts. 23,472 sentences by exact pattern match of head words of P-I-O categories. Afterward, the terms with top frequencies were used as the features of Naïve Bayesian classifier. This approach achieves F-measure values of 0.91 for Patient/Problem, 0.75 for Intervention and 0.88 for Outcome, comparable to previous studied based on mixed textural, paragraphical, and semantic features. In conclusion, we show that by stricter pattern matching criteria of training set, detection and classification of PICO elements can be reproducible with minimal expert intervention. The results of this work are higher than previous studies.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Citations
Cites
Evidence based medicine: what it is and what it isn't
Sackett, D. L., Rosenberg, W. M. C., Gray, J. A. M., Haynes, R. B., & Richardson, W. S. (1996) · BMJ · doi:10.1136/bmj.312.7023.71 · Cited by 15,769 · cited 1×
“Evidence-based medicine or practice (EBM, EBP) involves answering clinical questions by analysis of related articles from literature databases such as PubMed [1], [2].”
MethodUtilization of the PICO framework to improve searching PubMed for clinical questions
Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P. (2007) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-7-16 · Cited by 3,053
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · cited 1×
“…2) ACQUIRING (SEARCHING): applying a stepwise approach to judge the compatibility of the acquired literature to the present question on in the [3], then 3) APPRAISAL: using adequate critical appraisal tools based on the type of problems to judge the importance and validity of the literature.”
MethodAnswering Clinical Questions with Knowledge-Based and Statistical Techniques
Demner-Fushman, D., & Lin, J. (2007) · Computational Linguistics · doi:10.1162/coli.2007.33.1.63 · Cited by 296 · cited 2×
“Knowledge-based and statistical techniques were also used to identify and extract the PICO text of clinical questions [7]-[10].”
MethodCombining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131 · cited 7×
“We found that the performance of current method for P-and O-sentences detection is better than the previous study [5], provided that the feature selection ratio is greater than 3%.”
BackgroundMethodResultInfluentialCategorization of sentence types in medical abstracts.
McKnight, L., & Srinivasan, P. (2003) · PubMed · Cited by 126
Sentence retrieval for abstracts of randomized controlled trials.
Chung, G. Y. (2009) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-9-10 · Cited by 83
The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine
Lin, J., & Demner-Fushman, D. (2006) · Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1148170.1148191 · Cited by 59 · cited 1×
“Evidence-based medicine or practice (EBM, EBP) involves answering clinical questions by analysis of related articles from literature databases such as PubMed [1], [2].”
MethodDawes, M., Pluye, P., Shea, L., Grad, R., Greenberg, A., & Nie, J.-Y. (2007) · Journal of Innovation in Health Informatics · doi:10.14236/jhi.v15i1.640 · Cited by 47
Research works printed but not linked: 1
- demner-fushman, 2005, Knowledge Extraction for Clinical Question Answering: Preliminary Results, American Association for Artificial Intelligence
Cited by
PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical AbstractsPreprint
Dernoncourt, F., & Lee, J. Y. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1710.06071 · Cited by 320
Automating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
“…support vector machines (SVM), and multi-layer perceptron (MLP) Sentence Abstract F-score of 86.3 % for P, 67 % for I (and C), and 56.3 % for O Huang et al. (2011) [17] PICO (except C) 23,472 sentences from the structured abstracts naïve Bayes Sentence Abstract F-measure of 0.91 for…”
BackgroundInfluentialData extraction methods for systematic review (semi)automation: a living systematic review
Schmidt, L., Olorisade, B. K., McGuinness, L. A., Thomas, J., & Higgins, J. P. T. (2021) · F1000Research · doi:10.12688/f1000research.51117.1 · Cited by 106
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · F1000Research · doi:10.12688/f1000research.51117.2 · Cited by 44
Proceedings of the BioNLP 2018 workshop
(2018) · doi:10.18653/v1/w18-23 · Cited by 71
Afzal, M., Alam, F., Malik, K. M., & Malik, G. M. (2020) · Journal of Medical Internet Research · doi:10.2196/19810 · Cited by 65
“Subsequently, researchers have used the PICO structure for information retrieval and sentence identification in the text of biomedical documents [18-21].”
MethodPICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks.
Jin, D., & Szolovits, P. (2018) · Proceedings of the BioNLP 2018 workshop · doi:10.18653/v1/w18-2308 · Cited by 61
“, 2011; Chung, 2009; Chung and Coiera, 2007) and Multi-Layer Perceptron (MLP) (Boudin et al., 2010a; Huang et al., 2011).”
BackgroundMethodInfluentialPICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52
Self-citationAdvancing PICO element detection in biomedical text via deep neural networks.
Jin, D., & Szolovits, P. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa256 · Cited by 41
Stylianou, N., & Vlahavas, I. P. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103767 · Cited by 38
Data extraction methods for systematic review (semi)automation: Update of a living systematic review
Schmidt, L., Finnerty Mutlu, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2025) · F1000Research · doi:10.12688/f1000research.51117.3 · Cited by 27
“Finally, several publications discuss that a larger amount of benchmarking datasets could increase the comparability between published systems.(30,65,80) 4.”
AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs.
Ghosh, M., Mukherjee, S., Ganguly, A., Basuchowdhuri, P., Naskar, S. K., & Ganguly, D. (2024) · Methods · doi:10.1016/j.ymeth.2024.04.005 · Cited by 27
“According to earlier research, the entity extraction task from bio-medical literature was performed at the sentence level when a large number of annotated data instances were unavailable [46, 47].”
MethodAceso: PICO-Guided Evidence Summarization on Medical Literature.
Zhang, X., Geng, P., Zhang, T., Lu, Q., Gao, P., & Mei, J. (2020) · IEEE Journal of Biomedical and Health Informatics · doi:10.1109/jbhi.2020.2984704 · Cited by 25
“Many machine learning models have been proposed to detect PICO elements, including Naive Bayes [22], [23], Random Forest [22], Support Vector Machines [5], Conditional Random Fields [21] and multilayer perceptions [24].”
MethodStylianou, N., Razis, G., Goulis, D. G., & Vlahavas, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101949 · Cited by 22
Du, J., Wang, D., Lin, B., He, L., Huang, L., Wang, J., Manion, F. J., Li, Y., Cossrow, N., & Yao, L. (2025) · Scientific Reports · doi:10.1038/s41598-025-03979-5 · Cited by 21
“The majority of existing efforts have been focusing on sentence-level information extraction from the article abstract, e.g., recognizing PICO (Population, Intervention, Comparison, Outcomes) sentences from the abstract 8,9 .”
Trenta, A., Hunter, A., & Riedel, S. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.05209 · Cited by 15
Whitton, J., & Hunter, A. (2023) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2023.102661 · Cited by 20
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad036 · Cited by 14
An Automatic Method for Extracting Innovative Ideas Based on the Scopus® Database
Chen, L., & Fang, H. (2019) · KNOWLEDGE ORGANIZATION · doi:10.5771/0943-7444-2019-3-171 · Cited by 14
Emerald 110k: A Multidisciplinary Dataset for Abstract Sentence Classification
Stead, C., Smith, S., Busch, P. A., & Vatanasakdakul, S. (2019) · Cited by 11
Extracting PICO elements from RCT abstracts using 1-2gram analysis and multitask classification.
Yuan, X., Xiaoli, L., Shilei, L., Qinwen, S., & Ke, L. (2019) · Proceedings of the third International Conference on Medical and Health Informatics 2019 - ICMHI 2019 · doi:10.1145/3340037.3340043 · Cited by 10
“We designed three separate binary classification models for P, I and O classification to mitigate the possible effects of the imbalanced data distribution on the model classification results [23][24] and to avoid sentences with fuzzy labels being assigned as negative samples of non-P/I/O labels, which could affect the…”
InfluentialSystematic review automation methods
Norman, C. R. (2020) · theses.fr (ABES) · doi:10.70675/044ffa34z5c9bz433az99c8z64697dcd28a2 · Cited by 8
Indonesian medical question classification with pattern matching
Suwarningsih, W., Purwarianti, A., & Supriana, I. (2015) · International Conference on Automation, Cognitive Science, Optics, Micro Electro-Mechanical System, and Information Technology (ICACOMIT) · doi:10.1109/icacomit.2015.7440185 · Cited by 6
“[7] brought a number of rhetorical roles using a sequential framework with Conditional Random Fields (CRFs) to label PICO at the sentence level.”
MethodResultSequential short-text classification with neural networks
Dernoncourt, F. (2017) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 5
Dhrangadhariya, A., & Müller, H. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.34 · Cited by 3
“A decade of automatic PICO information extraction was limited to sentence-level due to the un-availability of entity-annotated corpora (Boudin et al., 2010; Huang et al., 2011, 2013; Wallace et al., 2016; Jin and Szolovits, 2018).”
A Scoping Review of Adopted Information Extraction Methods for RCTs.
Aletaha, A., Nemati-Anaraki, L., Keshtkar, A., Sedghi, S., Keramatfar, A., & Korolyova, A. (2023) · Medical Journal of The Islamic Republic of Iran · doi:10.47176/mjiri.37.95 · Cited by 2
“Fourteen studies explored the extraction of interventions and outcomes (13, 33-35, 37, 41, 48, 51, 53, 55-57, 59-62).”
BackgroundPICO Classification Using Domain-Specific Features.
Singh, S., & Sharan, A. (2024) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-99-8646-0_35 · Cited by 1
Yalçınkaya, A., Kjelmann, K. G., Gholinezhad, S., Rahbek, O., Kold, S., & Husum, H. (2025) · Journal of Orthopaedic Surgery and Research · doi:10.1186/s13018-025-06386-8 · Cited by 1
Schmidt, L., Finnerty, A. N., Elmore, R., Olorisade, B. K., Thomas, J., & Higgins, J. P. T. (2023) · UCL Discovery (University College London)
Haroutunian, L. (2022) · theses.fr (ABES) · doi:10.70675/a3de09d0z30faz4d04z846fz1d75d3f69a3a
Natural language processing to support evidence quality assessment of clinical literature
Hoang, K. L. (2023) · Illinois Digital Environment for Access to Learning and Scholarship (University of Illinois at Urbana-Champaign)
Jones, C. (2024) · Adelaide Research & Scholarship (AR&S) (University of Adelaide)
Metadata
- Authors
- Huang, K.-C., Liu, C. C.-H., Yang, S.-S., Xiao, F., Wong, J.-M., Liao, C.-C., & Chiang, I.-J.
- Affiliations
- National Taiwan University (TW)
- Department of Health (TW)
- Venue
- 2011 IEEE International Conference on Granular Computing
- Publisher
- IEEE
- Published
- 2011-11
- Pages
- 279-283
- OpenAlex
- W1969122915
- Cited by
- 26
- References
- 9
- Citations by year
- 2025: 3 · 2024: 2 · 2023: 4 · 2022: 2 · 2021: 2 · 2020: 4 · 2019: 3 · 2018: 2 · 2017: 1 · 2015: 2 · 2013: 1
- FWCI
- 1.17
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- IEEE International Conference on Granular Computing (GrC) (publishedVersion)
- Topics
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Advanced Text Analysis Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Text and Document Classification Technologies
- Keywords
- clinical trial reporting, text classification, PubMed abstracts, pattern matching, Naive Bayes classifier, text features
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.