The DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Utilization of the PICO framework to improve searching PubMed for clinical questions
Open accessAbstract
Background: Supporting 21 st century health care and the practice of evidence-based medicine (EBM) requires ubiquitous access to clinical information and to knowledge-based resources to answer clinical questions. Many questions go unanswered, however, due to lack of skills in formulating questions, crafting effective search strategies, and accessing databases to identify best levels of evidence. Methods: This randomized trial was designed as a pilot study to measure the relevancy of search results using three different interfaces for the PubMed search system. Two of the search interfaces utilized a specific framework called PICO, which was designed to focus clinical questions and to prompt for publication type or type of question asked. The third interface was the standard PubMed interface readily available on the Web. Study subjects were recruited from interns and residents on an inpatient general medicine rotation at an academic medical center in the US. Thirty-one subjects were randomized to one of the three interfaces, given 3 clinical questions, and asked to search PubMed for a set of relevant articles that would provide an answer for each question. The success of the search results was determined by a precision score, which compared the number of relevant or gold standard articles retrieved in a result set to the total number of articles retrieved in that set. Results: Participants using the PICO templates (Protocol A or Protocol B) had higher precision scores for each question than the participants who used Protocol C, the standard PubMed Web interface. (Question 1: A = 35%, B = 28%, C = 20%; Question 2: A = 5%, B = 6%, C = 4%; Question 3: A = 1%, B = 0%, C = 0%) 95% confidence intervals were calculated for the precision for each question using a lower boundary of zero. However, the 95% confidence limits were overlapping, suggesting no statistical difference between the groups. Conclusion: Due to the small number of searches for each arm, this pilot study could not demonstrate a statistically significant difference between the search protocols. However there was a trend towards higher precision that needs to be investigated in a larger study to determine if PICO can improve the relevancy of search results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The population, interventions, comparisons, outcomes, and study design (PICOS) framework [27] was followed to identify key study concepts in the research question a priori, and to facilitate the search process.”
In the citing paper
“The population, intervention, comparison, and outcome framework [25] was used to develop the search strategy, which was implemented following the PRISMA-P (Preferred Reporting Items for Systematic Review and Meta-Analyses Protocols) checklist [26].”
From The Effectiveness of Artificial Intelligence Conversational Agents in Health Care: Systematic Review (2020) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931
Evidence-based medicine : how to practice and teach EBM
Straus, S. E. (2005) · Cited by 1,367
Evaluation of PICO as a knowledge representation for clinical questions.
Huang, X., Lin, J., & Demner‐Fushman, D. (2006) · Europe PMC (PubMed Central) · Cited by 844
Obstacles to answering doctors' questions about patient care with evidence: qualitative study
Ely, J., Osheroff, J. A., Ebell, M. H., Chambliss, M. L., Vinson, D. C., Stevermer, J. J., & Pifer, E. A. (2002) · BMJ · doi:10.1136/bmj.324.7339.710 · Cited by 502
Green, M. L., & Ruff, T. R. (2005) · Academic Medicine · doi:10.1097/00001888-200502000-00016 · Cited by 246
Do Online Information Retrieval Systems Help Experienced Clinicians Answer Clinical Questions?
Westbrook, J. I. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1717 · Cited by 181
Residents’ medical information needs in clinic: are they being met?
Green, M. L., Ciampi, M., & Ellis, P. J. (2000) · The American Journal of Medicine · doi:10.1016/s0002-9343(00)00458-7 · Cited by 180
Effect of online literature searching on length of stay and patient care costs
Klein, M. S., Ross, F. V., Adams, D. L., & Gilbert, C. M. (1994) · Academic Medicine · doi:10.1097/00001888-199406000-00017 · Cited by 141
Robustness of empirical search strategies for clinical content in MEDLINE.
Wilczynski, N. L., & Haynes, R. B. (2002) · PubMed · Cited by 62
Research works printed but not linked: 1
- PubMed. [http://www.pubmed.gov]
Cited by
Sorted by citations
Items 101–160 of 160
Changing Behavior Using the Model of Action Phases
Keller, L., Gollwitzer, P. M., & Sheeran, P. (2020) · The Handbook of Behavior Change · doi:10.1017/9781108677318.006 · Cited by 147
Prevalence of tinnitus and hyperacusis in children and adolescents: a systematic review
Rosing, S. N., Schmidt, J. H., Wedderkopp, N., & Baguley, D. M. (2016) · BMJ Open · doi:10.1136/bmjopen-2015-010596 · Cited by 140
Pjrek, E., Friedrich, M.-E., Cambioli, L., Dold, M., Jäger, F., Komorowski, A., Lanzenberger, R., Kasper, S., & Winkler, D. (2020) · Psychotherapy and Psychosomatics · doi:10.1159/000502891 · Cited by 136
Khadra, D., Itani, L., Tannir, H., Kreidieh, D., Masri, D. E., & Ghoch, M. E. (2019) · World Journal of Diabetes · doi:10.4239/wjd.v10.i5.311 · Cited by 130
Cuthbert, C. A., Farragher, J. F., Hemmelgarn, B. R., Ding, Q., McKinnon, G. P., & Cheung, W. Y. (2019) · Psycho-Oncology · doi:10.1002/pon.5215 · Cited by 135
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
de Jong, C. C., Ros, W. J., & Schrijvers, G. (2014) · Journal of Medical Internet Research · doi:10.2196/jmir.3000 · Cited by 124
Dal Farra, F., Risio, R. G., Vismara, L., & Bergna, A. (2021) · Complementary Therapies in Medicine · doi:10.1016/j.ctim.2020.102616 · Cited by 123
Singh, B., Disipio, T., Peake, J., & Hayes, S. C. (2016) · Archives of Physical Medicine and Rehabilitation · doi:10.1016/j.apmr.2015.09.012 · Cited by 119
Systematic Review of Deep Learning Image Analyses for the Diagnosis and Monitoring of Skin Disease
Choy, S. P., Kim, B. J., Paolino, A., Tan, W. R., Lim, S. M. L., Seo, J., Tan, S. P., Francis, L., Tsakok, T., Simpson, M., et al. (2023) · npj Digital Medicine · doi:10.1038/s41746-023-00914-8 · Cited by 105
Lam, L. F., & Lawlis, T. R. (2017) · Clinical Nutrition · doi:10.1016/j.clnu.2016.06.013 · Cited by 91
Esketamine Treatment for Depression in Adults: A PRISMA Systematic Review and Meta-Analysis
Fountoulakis, K. N., Saitis, A., & Schatzberg, A. F. (2025) · American Journal of Psychiatry · doi:10.1176/appi.ajp.20240515 · Cited by 90
Le Morvan de Sequeira, C., Hengstberger, C., Enck, P., & Mack, I. (2022) · Nutrients · doi:10.3390/nu14030621 · Cited by 82
Beukes, E. W., Manchaiah, V., Allen, P. M., Baguley, D. M., & Andersson, G. (2019) · Trends in Hearing · doi:10.1177/2331216519851749 · Cited by 75
Hants, L., Bail, K., & Paterson, C. (2023) · Journal of Clinical Nursing · doi:10.1111/jocn.16823 · Cited by 79
Improving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
Reeves, R., Curran, D., Gleeson, A., & Hanna, D. (2022) · Behavior Modification · doi:10.1177/0145445521991102 · Cited by 69
“Utilizing the Participants, Interventions, Comparators, Outcomes (PICO) framework (Schardt et al., 2007) the present meta-analysis included studies where: Participants had elevated levels of PSA as identified through: (1) Participants scoring above the clinical cut-off on a PSA measure, for…”
MethodAssociation Between Asthma and Periodontal Disease: A Systematic Review and Meta-Analysis
Moraschini, V., Calasans‐Maia, J. D. A., & Calasans‐Maia, M. D. (2018) · Journal of Periodontology · doi:10.1902/jop.2017.170363 · Cited by 68
Resistance training variables for optimization of muscle hypertrophy: an umbrella review.
Bernárdez-Vázquez, R., Raya-González, J., Castillo, D., & Beato, M. (2022) · Frontiers in Sports and Active Living · doi:10.3389/fspor.2022.949021 · Cited by 67
PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks.
Jin, D., & Szolovits, P. (2018) · Proceedings of the BioNLP 2018 workshop · doi:10.18653/v1/w18-2308 · Cited by 61
Yin, Y.-H., Liu, J. Y. W., & Välimäki, M. (2020) · Experimental Gerontology · doi:10.1016/j.exger.2020.110937 · Cited by 61
Video games and their associations with physical health: a scoping review
Huard Pelletier, V., Lessard, A., Piché, F., Tétreau, C., & Descarreaux, M. (2020) · BMJ Open Sport & Exercise Medicine · doi:10.1136/bmjsem-2020-000832 · Cited by 61
Eating Disorders Spectrum During the COVID Pandemic: A Systematic Review
Miniati, M., Marzetti, F., Palagini, L., Marazziti, D., Orrù, G., Conversano, C., & Gemignani, A. (2021) · Frontiers in Psychology · doi:10.3389/fpsyg.2021.663376 · Cited by 61
Williams, T., McCaul, M., Schwarzer, G., Cipriani, A., Stein, D. J., & Ipser, J. (2020) · Acta Neuropsychiatrica · doi:10.1017/neu.2020.6 · Cited by 59
PICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52
Verbrugge, S. A. J., Schönfelder, M., Becker, L., Yaghoob Nezhad, F., Hrabě de Angelis, M., & Wackerhage, H. (2018) · Frontiers in Physiology · doi:10.3389/fphys.2018.00553 · Cited by 56
Giovannetti, A. M., Quintas, R., Tramacere, I., Giordano, A., Confalonieri, P., Messmer Uccelli, M., Solari, A., & Pakenham, K. I. (2020) · PLOS ONE · doi:10.1371/journal.pone.0231380 · Cited by 53
Treatment of provoked vulvodynia: a systematic review
Bohm-Starke, N., Ramsay, K. W., Lytsy, P., Nordgren, B., Sjöberg, I., Moberg, K., & Flink, I. (2022) · The Journal of Sexual Medicine · doi:10.1016/j.jsxm.2022.02.008 · Cited by 51
Bilgin, A., Muz, G., & Yuce, G. E. (2022) · Patient Education and Counseling · doi:10.1016/j.pec.2022.04.008 · Cited by 50
Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Tian, K., Shirui, Z., & Chunhua, W. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190209 · Cited by 45
Questions and answers about conducting systematic reviews in sport and exercise psychology
Gunnell, K., Poitras, V. J., & Tod, D. (2020) · International Review of Sport and Exercise Psychology · doi:10.1080/1750984x.2019.1695141 · Cited by 45
Chair-Based Exercise Interventions for Nursing Home Residents: A Systematic Review
Cordes, T., Schoene, D., Kemmler, W., & Wollesen, B. (2021) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2020.09.042 · Cited by 43
Advancing PICO element detection in biomedical text via deep neural networks.
Jin, D., & Szolovits, P. (2020) · Bioinformatics · doi:10.1093/bioinformatics/btaa256 · Cited by 41
Impact of PubMed search filters on the retrieval of evidence by physicians
Shariff, S. Z., Sontrop, J. M., Haynes, R. B., Iansavichus, A. V., McKibbon, K. A., Wilczynski, N. L., Weir, M. A., Speechley, M. R., Thind, A., & Garg, A. X. (2012) · Canadian Medical Association Journal · doi:10.1503/cmaj.101661 · Cited by 38
Barreto, G., Loureiro, L. M. R., Reis, C. E. G., & Saunders, B. (2023) · European Journal of Sport Science · doi:10.1080/17461391.2022.2049885 · Cited by 36
Chung, G. Y.-C. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.12.011 · Cited by 34
Gray, S. M., McKay, H. A., Nettlefold, L., Race, D., Macdonald, H. M., Naylor, P.-J., & Sims-Gould, J. (2021) · British Journal of Sports Medicine · doi:10.1136/bjsports-2020-102465 · Cited by 35
Greenwood, D. A., Litchman, M. L., Isaacs, D., Blanchette, J. E., Dickinson, J. K., Hughes, A., Colicchio, V. D., Ye, J., Yehl, K., Todd, A., et al. (2022) · Journal of Diabetes Science and Technology · doi:10.1177/19322968211036430 · Cited by 31
Gelashvili-Luik, T., Vihma, P., & Pappel, I. (2025) · Frontiers in Artificial Intelligence · doi:10.3389/frai.2025.1595930 · Cited by 30
Classification of PICO elements by text features systematically extracted from PubMed abstracts.
Huang, K.-C., Liu, C. C.-H., Yang, S.-S., Xiao, F., Wong, J.-M., Liao, C.-C., & Chiang, I.-J. (2011) · 2011 IEEE International Conference on Granular Computing · doi:10.1109/grc.2011.6122608 · Cited by 26
Klos, B., Cook, J., Crepaz, L., Weiland, A., Zipfel, S., & Mack, I. (2023) · European Journal of Nutrition · doi:10.1007/s00394-022-03054-z · Cited by 27
GPS use and navigation ability: A systematic review and meta-analysis
Miola, L., Muffato, V., Sella, E., Meneghetti, C., & Pazzaglia, F. (2024) · Journal of Environmental Psychology · doi:10.1016/j.jenvp.2024.102417 · Cited by 24
Munawar, K., Choudhry, F. R., Lee, S. H., Siau, C. S., Kadri, N. B. M., & Binti Sulong, R. M. (2021) · Heliyon · doi:10.1016/j.heliyon.2021.e07842 · Cited by 22
Yang, J.-M., Ye, H., Long, Y., Zhu, Q., Huang, H., Zhong, Y.-B., Luo, Y., Yang, L., & Wang, M.-Y. (2023) · Journal of Medical Internet Research · doi:10.2196/44000 · Cited by 22
Improving Medical Information Retrieval with PICO Element Detection.
Boudin, F., Shi, L., & Nie, J.-Y. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_8 · Cited by 21
Verbrugge, S. A. J., Alhusen, J. A., Kempin, S., Pillon, N. J., Rozman, J., Wackerhage, H., & Kleinert, M. (2022) · Journal of Cellular Biochemistry · doi:10.1002/jcb.30179 · Cited by 21
Chew, M. T., Chan, C., Kobayashi, S., Cheng, H. Y., Wong, T. M., & Nicholson, L. L. (2023) · Pain Practice · doi:10.1111/papr.13227 · Cited by 18
Kang, T., Sun, Y., Kim, J. H., Ta, C., Perotte, A., Schiffer, K., Wu, M., Zhao, Y., Moustafa-Fahmy, N., Peng, Y., et al. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad036 · Cited by 14
Pinheiro, É. P., Cavalheiro do Espírito Santo, R., Peterson dos Santos, L., Gonçalves, W. V., Forgiarini Junior, L. A., Xavier, R. M., & Filippin, L. I. (2022) · Journal of the American Medical Directors Association · doi:10.1016/j.jamda.2022.06.009 · Cited by 16
Wong, M. M. H., Yuen-Man Chan, M., Ng, T. P., & Louie, J. C. Y. (2024) · Diabetes & Metabolic Syndrome: Clinical Research & Reviews · doi:10.1016/j.dsx.2024.102941 · Cited by 14
Systematic review on use of health incentives in U.S. to change maternal health behavior
Washio, Y., Atreyapurapu, S., Hayashi, Y., Taylor, S., Chang, K., Ma, T., & Isaacs, K. (2021) · Preventive Medicine · doi:10.1016/j.ypmed.2021.106442 · Cited by 13
Giovannetti, A. M., Pakenham, K. I., Presti, G., Quartuccio, M. E., Confalonieri, P., Bergamaschi, R., Grobberio, M., Di Filippo, M., Micheli, M., Brichetto, G., et al. (2022) · PLoS ONE · doi:10.1371/journal.pone.0267245 · Cited by 11
Extracting PICO elements from RCT abstracts using 1-2gram analysis and multitask classification.
Yuan, X., Xiaoli, L., Shilei, L., Qinwen, S., & Ke, L. (2019) · Proceedings of the third International Conference on Medical and Health Informatics 2019 - ICMHI 2019 · doi:10.1145/3340037.3340043 · Cited by 10
Öcalan, S., & Üzar‐Özçetin, Y. S. (2021) · Journal of Advanced Nursing · doi:10.1111/jan.14916 · Cited by 10
Oteri, V., Volpe, S., Lopes, M., Sceusa, G., Tumminia, A., Belfiore, A., Frasca, F., & Gullo, D. (2024) · Endocrine · doi:10.1007/s12020-024-04016-7 · Cited by 9
Procedures for transferring organizational knowledge during generational change: A systematic review
Igoa-Iraola, E., & Díez, F. (2024) · Heliyon · doi:10.1016/j.heliyon.2024.e27092 · Cited by 11
Tool for filtering PubMed search results by sample size.
Baladrón, C., Santos-Lozano, A., Aguiar, J. M., Lucia, A., & Martín-Hernández, J. (2018) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocx155 · Cited by 6
MacDonald, H., & Bézaire, V. (2024) · Advances in Physiology Education · doi:10.1152/advan.00171.2023 · Cited by 3
Kousios, A., Hadjivasilis, A., Kouis, P., & Panayiotou, A. (2020) · doi:10.21203/rs.3.rs-104160/v1 · Cited by 1
Diabetic gastroparesis: pathophysiology and impact on insulin timing choices
Galeano, F., Tumminia, A., Oteri, V., Le Moli, R., Piticchio, T., Scilletta, S., Di Marco, M., Di Pino, A., & Frasca, F. (2026) · Endocrine · doi:10.1007/s12020-026-04693-6
Metadata
- Authors
- Schardt, C., Adams, M. B., Owens, T., Keitz, S., & Fontelo, P.
- Affiliations
- Duke University (US)
- Miami VA Healthcare System (US)
- United States National Library of Medicine (US)
- Corresponding author
- Schardt, Connie
- Adams, Martha B
- Owens, Thomas
- Keitz, Sheri
- Fontelo, Paul
- Venue
- BMC Medical Informatics and Decision Making
- Published
- 2007-06-15
- Volume
- 7
- Issue
- 1
- Pages
- 16-16
- ISSN
- 1472-6947 (ISSN-L)
- PMC
- PMC1904193
- OpenAlex
- W2111722073
- Cited by
- 3,053
- References
- 14
- Citations by year
- 2026: 346 · 2025: 460 · 2024: 364 · 2023: 345 · 2022: 380 · 2021: 302 · 2020: 310 · 2019: 134 · 2018: 93 · 2017: 79 · 2016: 57 · 2015: 46 · 2014: 35 · 2013: 18 · 2012: 24
- FWCI
- 24.35
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- BMC Medical Informatics and Decision Making (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- Topics
- Health Sciences › Health Professions › General Health Professions › Health Sciences Research and Education
- Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
- Social Sciences › Decision Sciences › Statistics, Probability and Uncertainty › Meta-analysis and systematic reviews
- Keywords
- PICO framework, PubMed search, clinical question formulation, evidence-based medicine, search result relevance, search interface design, medical information retrieval
- MeSH terms
- Humans, Internship and Residency, Medical Informatics, Probability, Surveys and Questionnaires, Sensitivity and Specificity, MEDLINE, Information Storage and Retrieval, Evidence-Based Medicine, PubMed, Medical Subject Headings
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
