The DOI matches its Crossref record; 52 of the 52 references it cites were found in a registry or an index.
Answering Clinical Questions with Knowledge-Based and Statistical Techniques
Open accessAbstract
The combination of recent developments in question-answering research and the availability of unparalleled resources developed specifically for automatic semantic processing of text in the medical domain provides a unique opportunity to explore complex question answering in the domain of clinical medicine. This article presents a system designed to satisfy the information needs of physicians practicing evidence-based medicine. We have developed a series of knowledge extractors, which employ a combination of knowledge-based and statistical techniques, for automatically identifying clinically relevant aspects of MEDLINE abstracts. These extracted elements serve as the input to an algorithm that scores the relevance of citations with respect to structured representations of information needs, in accordance with the principles of evidence-based medicine. Starting with an initial list of citations retrieved by PubMed, our system can bring relevant abstracts into higher ranking positions, and from these abstracts generate responses that directly answer physicians' questions. We describe three separate evaluations: one focused on the accuracy of the knowledge extractors, one conceptualized as a document reranking task, and finally, an evaluation of answers by two physicians. Experiments on a collection of real-world clinical questions show that our approach significantly outperforms the already competitive PubMed baseline.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Baeza-Yates, R. A., & Ribeiro-Neto, B. A. (1999) · Cited by 11,505
Evidence-Based Medicine: How to Practice and Teach EBM
Sackett, D. L. (2000) · Cited by 5,950
A Comparative Study on Feature Selection in Text Categorization
Yang, Y., & Pedersen, J. (1997) · Cited by 4,790
The Journal of Family Practice
(1997) · The American Journal of Family Therapy · doi:10.1080/01926189708251066 · Cited by 3,067
The well-built clinical question: A key to evidence-based decisions
(1995) · ACP Journal Club · doi:10.7326/acpjc-1995-123-3-a12 · Cited by 2,931 · cited 3×
“The following four components have been iden-tified as the key elements of a question related to patient care (Richardson et al. 1995): (cid:1) What is the primary problem or disease?”
BackgroundMethodInfluentialThe Unified Medical Language System
Humphreys, B. L., McCray, A. T., & Lindberg, D. A. B. (1993) · Methods of Information in Medicine · doi:10.1055/s-0038-1634945 · Cited by 1,182
Evaluation of PICO as a knowledge representation for clinical questions.
Huang, X., Lin, J., & Demner‐Fushman, D. (2006) · Europe PMC (PubMed Central) · Cited by 898
Self-citationRELEVANCE: A review of and a framework for the thinking on the notion in information science
Saracevic, T. (1975) · Journal of the American Society for Information Science · doi:10.1002/asi.4630260604 · Cited by 863
Information Needs in Office Practice: Are They Being Met?
COVELL, D. G., UMAN, G. C., & MANNING, P. R. (1985) · Annals of Internal Medicine · doi:10.7326/0003-4819-103-4-596 · Cited by 854
Developing Optimal Search Strategies for Detecting Clinically Sound Studies in MEDLINE
Haynes, R. B., Wilczynski, N., McKibbon, K. A., Walker, C. J., & Sinclair, J. C. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95153434 · Cited by 765
Issues in Stacked Generalization
Ting, K. M., & Witten, I. H. (1999) · Journal of Artificial Intelligence Research · doi:10.1613/jair.594 · Cited by 576
Ebell, M. H., Siwek, J., Weiss, B. D., Woolf, S. H., Susman, J., Ewigman, B., & Bowman, M. (2004) · The Journal of the American Board of Family Medicine · doi:10.3122/jabfm.17.1.59 · Cited by 561
Evaluating Content Selection in Summarization: The Pyramid Method
Nenkova, A., & Passonneau, R. (2004) · Columbia University · doi:10.7916/d80r9xvd · Cited by 527
Analysis of questions asked by family doctors regarding patient care
Ely, J. W., Osheroff, J. A., Ebell, M. H., Bergus, G. R., Levy, B. T., Chambliss, M. L., & Evans, E. R. (1999) · BMJ · doi:10.1136/bmj.319.7206.358 · Cited by 522 · cited 7×
“All questions were manually classified into one of the four clinical tasks; the distribution of the questions roughly follows the prevalence of each task type as observed in natural settings, noted by Ely et al. (1999). The final step in the preparation process was manual translation of the natural language questions…”
BackgroundMethodInfluentialIssues in Stacked Generalization
Ting, K. M., & Witten, I. H. (1999) · Cited by 521
Rindflesch, T. C., & Fiszman, M. (2003) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2003.11.003 · Cited by 518
American journal of diseases of children
(1993) · Journal of the American Dietetic Association · doi:10.1016/0002-8223(93)91837-g · Cited by 484
Feature Selection for Unbalanced Class Distribution and Naive Bayes
Mladenić, D., & Grobelnik, M. (1999) · Cited by 427
Answering Physicians' Clinical Questions: Obstacles and Potential Solutions
Ely, J. W., Osheroff, J. A., Chambliss, M. L., Ebell, M. H., & Rosenbaum, M. E. (2005) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1608 · Cited by 388
Annual Review of Information Science and Technology 1998
Williams, M. E. (1999) · Cited by 334
Senie, R. T., Rosen, P. P., Rhodes, P., Lesser, M. L., & Kinne, D. W. (1992) · Rehabilitation Oncology · doi:10.1097/01893697-199210010-00023 · Cited by 300
(2001) · PubMed · Cited by 275
Users' criteria for relevance evaluation: A cross-situational comparison
Barry, C. L., & Schamber, L. (1998) · Information Processing & Management · doi:10.1016/s0306-4573(97)00078-2 · Cited by 272
Gutzwiller, F., & Paccaud, F. (1987) · PubMed · Cited by 240
The NLM Indexing Initiative's Medical Text Indexer.
Aronson, A. R., Mork, J. G., Gay, C. W., Humphrey, S. M., & Rogers, W. J. (2004) · PubMed · Cited by 206
Measuring use patterns of online journals and databases.
Groote, S. L. D., & Dorsch, J. L. (2003) · Europe PMC (PubMed Central) · Cited by 161
The Language of Bioscience: Facts, Speculations, and Statements in Between
Light, M., Qiu, X., & Srinivasan, P. (2004) · North American Chapter of the Association for Computational Linguistics · Cited by 161
Can primary care physicians' questions be answered using the medical journal literature?
Gorman, P., Ash, J. S., & Wykoff, L. W. (1994) · PubMed Central · Cited by 156
Categorization of sentence types in medical abstracts.
McKnight, L., & Srinivasan, P. (2003) · PubMed · Cited by 126
Abstraction summarization for managing the biomedical research literature
Marcelo Fiszman, B., Rindflesch, T. C., & Halil Kilicoglu, B. (2004) · doi:10.3115/1596431.1596442 · Cited by 119
Answer extraction, semantic clustering, and extractive summarization for clinical question answering
Demner‐Fushman, D., & Lin, J. (2006) · International Conference on Computational Linguistics · doi:10.3115/1220175.1220281 · Cited by 117
Self-citationChambliss, M. L., & Conley, J. (1996) · PubMed · Cited by 111
What Makes a Good Answer? The Role of Context in Question Answering.
Lin, J., Quan, D., Sinha, V., Bakshi, K., Huynh, D. F., Katz, B., & Karger, D. R. (2003) · Cited by 102
Self-citationWilczynski, N. L., McKibbon, K. A., & Haynes, R. B. (2001) · PubMed · Cited by 101
Exploiting a large thesaurus for information retrieval
Aronson, A. R., Rindflesch, T. C., & Browne, A. C. (1994) · Multimedia Information Retrieval · Cited by 96
A study of biomedical concept identification: MetaMap vs. people.
Pratt, W., & Yetisgen-Yildiz, M. (2003) · PubMed · Cited by 94
Customization in a unified framework for summarizing medical literature
Elhadad, N., Kan, M., Klavans, J. L., & McKeown, K. R. (2004) · Artificial Intelligence in Medicine · doi:10.1016/j.artmed.2004.07.018 · Cited by 94
Towards a Medical Question-Answering System: a Feasibility Study
Jacquemart, P., & Zweigenbaum, P. (2003) · Studies in health technology and informatics · doi:10.3233/978-1-60750-939-4-463 · Cited by 82
Answering Definition Questions Using Multiple Knowledge Sources
Hildebrandt, W., Katz, B., & Lin, J. (2004) · Cited by 82
Self-citationWill Pyramids Built of Nuggets Topple Over?
Lin, J., & Demner-Fushman, D. (2006) · Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics - · doi:10.3115/1220835.1220884 · Cited by 77
Self-citationAutomatically evaluating answers to definition questions
Lin, J., & Demner‐Fushman, D. (2005) · doi:10.3115/1220575.1220692 · Cited by 73
Self-citationGenerative content models for structural analysis of medical abstracts
Lin, J., Karakos, D., Demner‐Fushman, D., & Khudanpur, S. P. (2006) · Workshop on Biomedical Language Processing (BioNLP) · doi:10.3115/1567619.1567631 · Cited by 71
Self-citationAnalysis of Semantic Classes in Medical Text for Question Answering
Niu, Y., & Hirst, G. (2004) · Cited by 62
The role of knowledge in conceptual retrieval: a study in the domain of clinical medicine
Lin, J., & Demner‐Fushman, D. (2006) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/1148170.1148191 · Cited by 60
Self-citationAutomatically extracting information needs from complex clinical questions
Cao, Y.-G., Cimino, J. J., Ely, J., & Yu, H. Q. (2010) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2010.07.007 · Cited by 53
Tbahriti, I., CHICHESTER, C., Lisacek, F., & Ruch, P. (2005) · International Journal of Medical Informatics · doi:10.1016/j.ijmedinf.2005.06.007 · Cited by 52
Knowledge-based Methods to Help Clinicians Find Answers in MEDLINE
Sneiderman, C. A., Demner‐Fushman, D., Marcelo Fiszman, B., Ide, N., & Rindflesch, T. C. (2007) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m2407 · Cited by 46
Self-citationAutomatically Identifying Health Outcome Information in MEDLINE Records
Demner-Fushman, D., Few, B., Hauser, S. E., & Thoma, G. (2006) · Journal of the American Medical Informatics Association · doi:10.1197/jamia.m1911 · Cited by 35 · cited 11×
“However, studies have shown that existing systems for searching MEDLINE (such as PubMed, the search service provided by the National Library of Medicine) are often inadequate and unable to supply clinically relevant answers in a timely manner (Gorman, Ash, and Wykoff 1994; Chambliss and Conley 1996).”
BackgroundMethodInfluentialSelf-citationEvaluating Summaries and Answers: Two Sides of the Same Coin?
Lin, J., & Demner‐Fushman, D. (2005) · Cited by 11
Self-citation(2001) · doi:10.1002/pds.4850 · Cited by 7
Schlick, M. (1974) · LEP. Library of exact philosophy · doi:10.1007/978-3-7091-3099-5_22 · Cited by 7
Semantic characteristics of MEDLINE citations useful in therapeutic decision-making.
Sneiderman, C. A., Demner‐Fushman, D., Marcelo Fiszman, B., & Rindflesch, T. C. (2005) · PubMed Central · Cited by 3
Self-citation
Cited by
What can natural language processing do for clinical decision support?
Demner-Fushman, D., Chapman, W. W., & McDonald, C. J. (2009) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2009.08.007 · Cited by 688
Self-citationAutomating data extraction in systematic reviews: a systematic review
Jonnalagadda, S. R., Goyal, P., & Huffman, M. D. (2015) · Systematic Reviews · doi:10.1186/s13643-015-0066-7 · Cited by 287
Frontiers of biomedical text mining: current progress
Zweigenbaum, P., Demner-Fushman, D., Yu, H., & Cohen, K. B. (2007) · Briefings in Bioinformatics · doi:10.1093/bib/bbm045 · Cited by 282
Self-citationCan ChatGPT Write a Good Boolean Query for Systematic Review Literature Search?
Wang, S., Scells, H., Koopman, B., & Zuccon, G. (2023) · Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval · doi:10.1145/3539618.3591703 · Cited by 275
“Automated methods have been developed for query formulation [13, 34, 55], query refinement [34, 60] and query exploitation [7, 17].”
MethodPredicting information seeker satisfaction in community question answering
Liu, Y., Bian, J., & Agichtein, E. (2008) · Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/1390334.1390417 · Cited by 254
Metadata
- Authors
- Demner-Fushman, D., & Lin, J.
- Venue
- Computational Linguistics
- Publisher
- MIT Press - Journals
- Published
- 2007-03
- Volume
- 33
- Issue
- 1
- Pages
- 63-103
- ISSN
- 0891-2017 (ISSN-L)
- 1530-9312
- OpenAlex
- W2134568263
- Cited by
- 296
- References
- 52
- FWCI
- 5.51
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.