The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Answering Clinical Questions with Knowledge-Based and Statistical Techniques
Open accessAbstract
The combination of recent developments in question-answering research and the availability of unparalleled resources developed specifically for automatic semantic processing of text in the medical domain provides a unique opportunity to explore complex question answering in the domain of clinical medicine. This article presents a system designed to satisfy the information needs of physicians practicing evidence-based medicine. We have developed a series of knowledge extractors, which employ a combination of knowledge-based and statistical techniques, for automatically identifying clinically relevant aspects of MEDLINE abstracts. These extracted elements serve as the input to an algorithm that scores the relevance of citations with respect to structured representations of information needs, in accordance with the principles of evidence-based medicine. Starting with an initial list of citations retrieved by PubMed, our system can bring relevant abstracts into higher ranking positions, and from these abstracts generate responses that directly answer physicians' questions. We describe three separate evaluations: one focused on the accuracy of the knowledge extractors, one conceptualized as a document reranking task, and finally, an evaluation of answers by two physicians. Experiments on a collection of real-world clinical questions show that our approach significantly outperforms the already competitive PubMed baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
A Comparative Study on Feature Selection in Text Categorization
Yang, Y., & Pedersen, J. (1997) · Cited by 4,790
The Journal of Family Practice
(1997) · The American Journal of Family Therapy · doi:10.1080/01926189708251066 · Cited by 3,067
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · cited 3×
“The following four components have been iden-tified as the key elements of a question related to patient care (Richardson et al. 1995): (cid:1) What is the primary problem or disease?”
BackgroundMethodInfluentialThe Unified Medical Language System
Humphreys, B. L., McCray, A. T., & Lindberg, D. A. B. (1993) · Methods of Information in Medicine · doi:10.1055/s-0038-1634945 · Cited by 1,182
Evaluation of PICO as a knowledge representation for clinical questions.
Huang, X., Lin, J., & Demner‐Fushman, D. (2006) · Europe PMC (PubMed Central) · Cited by 898
Self-citationRELEVANCE: A review of and a framework for the thinking on the notion in information science
Saracevic, T. (1975) · Journal of the American Society for Information Science · doi:10.1002/asi.4630260604 · Cited by 863
Information Needs in Office Practice: Are They Being Met?
COVELL, D. G., UMAN, G. C., & MANNING, P. R. (1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-596 · Cited by 854
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N., McKibbon, K. A., Walker, C. J., & Sinclair, J. C. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 765
Cited by
What can natural language processing do for clinical decision support?
Demner-Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 688
Self-citationAutomating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
Frontiers of biomedical text mining: current progress
Zweigenbaum, P., Demner-Fushman, D., Yu, H., & Cohen, K. B. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbm045 · Cited by 282
Self-citationCan ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275
“Automated methods have been developed for query formulation [13, 34, 55], query refinement [34, 60] and query exploitation [7, 17].”
MethodPredicting information seeker satisfaction in community question answering
Liu, Y., Bian, J., & Agichtein, E. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390417 · Cited by 254
Nye, B., Li, J. J., Patel, R., Yang, Y., Marshall, I., Nenkova, A., & Wallace, B. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1019 · Cited by 224
Automatic classification of sentences to support Evidence Based Medicine.
Kim, S. N., Martinez, D., Cavedon, L., & Yencken, L. (2011) · BMC Bioinformatics · doi:10.1186/1471-2105-12-s2-s5 · Cited by 214
“It is important to note that this is the only previous work in the literature that uses the Other tag as we do. Demner-Fushman and Lin [ 7 ] also applied their final PICO classifiers to a novel weighting formula for medical information retrieval (IR), significantly improving the baseline for the task.”
BackgroundMethodInfluential(2016) · PubMed · Cited by 134
Combining classifiers for robust PICO element detection.
Boudin, F., Nie, J.-Y., Bartlett, J. C., Grad, R., Pluye, P., & Dawes, M. (2010) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-10-29 · Cited by 131
Automating Biomedical Evidence Synthesis: RobotReviewer
Marshall, I., Kuiper, J., Banner, E., & Wallace, B. C. (2017) · Proceedings of ACL 2017, System Demonstrations · doi:10.18653/v1/p17-4002 · Cited by 96
Improving reference prioritisation with PICO recognition.
Brockmeier, A. J., Ju, M., Przybyła, P., & Ananiadou, S. (2019) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-019-0992-8 · Cited by 62
Bui, D. D. A., Del Fiol, G., Hurdle, J. F., & Jonnalagadda, S. (2016) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2016.10.014 · Cited by 70
“Demner-Fushman and Lin (13), Kelly and Yang (14), and Hansen et al.”
BackgroundAfzal, M., Alam, F., Malik, K. M., & Malik, G. M. (2020) · Journal of Medical Internet Research · doi:10.2196/19810 · Cited by 65
“PICO-based ATS remains an unexplored area; however, work has been done on individual aspects such as PICO elements identification in the text [21,29,30], quality recognition of therapy-related biomedical studies [40,41], and sentence-based summarization without PICO and quality evaluation [3,32].”
MethodIdentifying scientific artefacts in biomedical literature: The Evidence Based Medicine use case.
Hassanzadeh, H., Groza, T., & Hunter, J. (2014) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2014.02.006 · Cited by 62
PICO Element Detection in Medical Text via Long Short-Term Memory Neural Networks.
Jin, D., & Szolovits, P. (2018) · Proceedings of the BioNLP 2018 workshop · doi:10.18653/v1/w18-2308 · Cited by 61
Biomedical text mining for research rigor and integrity: tasks, challenges, directions
Kilicoglu, H. (2017) · Briefings in Bioinformatics · doi:10.1093/bib/bbx057 · Cited by 61
Kaufman, D. R., Sheehan, B., Stetson, P., Bhatt, A. R., Field, A. I., Patel, C., & Maisel, J. M. (2016) · JMIR Medical Informatics · doi:10.2196/medinform.5544 · Cited by 58
PICO element detection in medical text without metadata: are first sentences enough?
Huang, K.-C., Chiang, I.-J., Xiao, F., Liao, C.-C., Liu, C. C.-H., & Wong, J.-M. (2013) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2013.07.009 · Cited by 52
Automatic Summarization of Results from Clinical Trials
Summerscales, R. L., Argamon, S., Bai, S., Hupert, J., & Schwartz, A. (2011) · 2011 IEEE International Conference on Bioinformatics and Biomedicine · doi:10.1109/bibm.2011.72 · Cited by 49
Pretraining to Recognize PICO Elements from Randomized Controlled Trial Literature.
Tian, K., Shirui, Z., & Chunhua, W. (2019) · Studies in Health Technology and Informatics · doi:10.3233/shti190209 · Cited by 45
Evaluation of ChatGPT as a diagnostic tool for medical learners and clinicians
Hadi, A., Tran, E., Nagarajan, B., & Kirpalani, A. (2024) · PLOS ONE · doi:10.1371/journal.pone.0307383 · Cited by 45
Medical Relation Extraction with Manifold Models
Wang, C., & Fan, J. (2014) · Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.3115/v1/p14-1078 · Cited by 40
Chung, G. Y.-C. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2008.12.011 · Cited by 34
Toward assessing clinical trial publications for reporting transparency
Kilicoglu, H., Rosemblat, G., Hoang, L., Wadhwa, S., Peng, Z., Malički, M., Schneider, J., & ter Riet, G. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2021.103717 · Cited by 34
Mining characteristics of epidemiological studies from Medline: a case study in obesity.
Karystianis, G., Buchan, I., & Nenadic, G. (2014) · Journal of Biomedical Semantics · doi:10.1186/2041-1480-5-22 · Cited by 23
“In clinical text mining, several attempts have been made to extract various kinds of information from case studies and clinical trials in particular [1-4,15-23].”
BackgroundClassification of PICO elements by text features systematically extracted from PubMed abstracts.
Huang, K.-C., Liu, C. C.-H., Yang, S.-S., Xiao, F., Wong, J.-M., Liao, C.-C., & Chiang, I.-J. (2011) · 2011 IEEE International Conference on Granular Computing · doi:10.1109/grc.2011.6122608 · Cited by 26
Stylianou, N., Razis, G., Goulis, D. G., & Vlahavas, I. (2020) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2020.101949 · Cited by 22
Hu, Y., Keloth, V. K., Raja, K., Chen, Y., & Xu, H. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad542 · Cited by 21
“…works were based on rule-based approaches and machine learning (ML) approaches (e.g. Support Vector Machine, Random Forest, Conditional Random Field) and rely heavily on hand-crafted features (Huang et al. 2006, Demner-Fushman and Lin 2007, Boudin et al. 2010, Chabou and Iglewski 2015, 2018).”
MethodImproving Medical Information Retrieval with PICO Element Detection.
Boudin, F., Shi, L., & Nie, J.-Y. (2010) · Lecture Notes in Computer Science · doi:10.1007/978-3-642-12275-0_8 · Cited by 21
Combination of conditional random field with a rule based method in the extraction of PICO elements.
Chabou, S., & Iglewski, M. (2018) · BMC Medical Informatics and Decision Making · doi:10.1186/s12911-018-0699-2 · Cited by 19
Jiang, L., Lan, M., Menke, J. D., Vorland, C. J., & Kilicoglu, H. (2024) · Scientific Reports · doi:10.1038/s41598-024-72130-7 · Cited by 18
“Most NLP research on RCT publications has focused on classifying sentences using the PICO framework (Population, Intervention, Comparator, Outcome) to aid the systematic review process and evidence-based medicine 27–34 .”
MethodExploiting classification correlations for the extraction of evidence-based practice information.
(2012) · PubMed Central · Cited by 16
Extracting PICO elements from RCT abstracts using 1-2gram analysis and multitask classification.
Yuan, X., Xiaoli, L., Shilei, L., Qinwen, S., & Ke, L. (2019) · Proceedings of the third International Conference on Medical and Health Informatics 2019 - ICMHI 2019 · doi:10.1145/3340037.3340043 · Cited by 10
Sanchez-Graillet, O., Witte, C., Grimm, F., & Cimiano, P. (2022) · Journal of Biomedical Semantics · doi:10.1186/s13326-022-00271-7 · Cited by 9
PICO Extraction by combining the robustness of machine-learning methods with the rule-based methods.
Chabou, S., & Iglewski, M. (2015) · 2015 World Congress on Information Technology and Computer Applications (WCITCA) · doi:10.1109/wcitca.2015.7367038 · Cited by 7
Evaluating automated entity extraction with respect to drug and non-drug treatment strategies.
Guo, J., Blake, C., & Guan, Y. (2019) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2019.103177 · Cited by 8
Intra-Template Entity Compatibility based Slot-Filling for Clinical Trial Information Extraction.
Witte, C., & Cimiano, P. (2022) · Proceedings of the 21st Workshop on Biomedical Language Processing · doi:10.18653/v1/2022.bionlp-1.18 · Cited by 5
“Then the TA module assignes these identifies SFCs to template instances of type Outcome, together with other SCFs extracted from other sentences.”
BackgroundInfluentialPICO Classification Using Domain-Specific Features.
Singh, S., & Sharan, A. (2024) · Lecture Notes in Electrical Engineering · doi:10.1007/978-981-99-8646-0_35 · Cited by 1
Diagnosing the limits of large language models for structured scientific evidence extraction
Tan, Z., D’Souza, J., & Auer, S. (2026) · International Journal on Digital Libraries · doi:10.1007/s00799-026-00449-9
Metadata
- Authors
- Demner-Fushman, D., & Lin, J.
- Venue
- Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2007-03
- Volume
- 33
- Issue
- 1
- Pages
- 63-103
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W2134568263
- Cited by
- 296
- References
- 52
- FWCI
- 5.51
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.