CoChat
Sign Up
Cited by 20Machine Learning in HealthcareInformation Retrieval and Search BehaviorArtificial Intelligence in Healthcare and Education

Retrieval-Based Diagnostic Decision Support: Mixed Methods Study

Open access
Abdullahi, Tassallah, Mercurio, Laura, Singh, Ritambhara, Eickhoff, Carsten·2024·JMIR Medical Informatics
DOI 10.2196/50209PMC PMC11222760

Published version of a preprint

Open this paper in CoChat — free accountOpen PDF

Abstract

BACKGROUND: Diagnostic errors pose significant health risks and contribute to patient mortality. With the growing accessibility of electronic health records, machine learning models offer a promising avenue for enhancing diagnosis quality. Current research has primarily focused on a limited set of diseases with ample training data, neglecting diagnostic scenarios with limited data availability. OBJECTIVE: This study aims to develop an information retrieval (IR)-based framework that accommodates data sparsity to facilitate broader diagnostic decision support. METHODS: We introduced an IR-based diagnostic decision support framework called CliniqIR. It uses clinical text records, the Unified Medical Language System Metathesaurus, and 33 million PubMed abstracts to classify a broad spectrum of diagnoses independent of training data availability. CliniqIR is designed to be compatible with any IR framework. Therefore, we implemented it using both dense and sparse retrieval approaches. We compared CliniqIR's performance to that of pretrained clinical transformer models such as Clinical Bidirectional Encoder Representations from Transformers (ClinicalBERT) in supervised and zero-shot settings. Subsequently, we combined the strength of supervised fine-tuned ClinicalBERT and CliniqIR to build an ensemble framework that delivers state-of-the-art diagnostic predictions. RESULTS: On a complex diagnosis data set (DC3) without any training data, CliniqIR models returned the correct diagnosis within their top 3 predictions. On the Medical Information Mart for Intensive Care III data set, CliniqIR models surpassed ClinicalBERT in predicting diagnoses with <5 training samples by an average difference in mean reciprocal rank of 0.10. In a zero-shot setting where models received no disease-specific training, CliniqIR still outperformed the pretrained transformer models with a greater mean reciprocal rank of at least 0.10. Furthermore, in most conditions, our ensemble framework surpassed the performance of its individual components, demonstrating its enhanced ability to make precise diagnostic predictions. CONCLUSIONS: Our experiments highlight the importance of IR in leveraging unstructured knowledge resources to identify infrequently encountered diagnoses. In addition, our ensemble framework benefits from combining the complementary strengths of the supervised and retrieval-based models to diagnose a broad spectrum of diseases.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.2196/50209

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • [20] CliniqIR showed that retrieval-based decision support outpaced Clinical BERT for rare conditions, mitigating long-tail distribution obstacles.

    “[20] CliniqIR showed that retrieval-based decision support outpaced Clinical BERT for rare conditions, mitigating long-tail distribution obstacles that traditional machine learning models cannot accurately address.”

    From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [20]

    The Results (‘Performance on MIMIC-III’) report that CliniqIR outperformed ClinicalBERT for diagnoses with up to three training samples, supporting the rare-condition comparison. The Introduction (‘Objectives’) describes limitations of supervised models on highly imbalanced data, but the source does not establish the broader claim that traditional machine-learning models cannot accurately address long-tail distributions; it also reports benefits from combining retrieval and supervised models. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • The importance of authentic clinician validation as an ethical principle may be lost on resource-constrained settings facing fast digitisation challenges.

    “The importance of authentic clinician validation as an ethical principle may be lost on resource-constrained settings facing the fast digitisation challenges [20].”

    From Multi-Agent Retrieval Augmented Generation for Clinical Decision Support: A Systematic Review and Integrative Conceptual Framework (2026) · cites this paper as [20]

    The Discussion’s Principal Findings emphasize clinicians’ ability to understand and trust diagnostic tools and describe collaboration with health professionals. The Limitations section says the approach may be valuable where computing resources are limited. Neither section describes clinician validation as an ethical principle or says its importance may be overlooked amid rapid digitisation. (LLM, conf=0.78)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 48

  1. [1]Barnett GO Cimino JJ Hupp JA Hoffer EP DXplain: an evolving diagnostic decision-support system JAMA 1987 07 03 258 1 67 74 10.1001/jama.1987.03400010071030 3295316 · DOI
  2. [2]Khoong EC Nouri SS Tuot DS Nundy S Fontil V Sarkar U Comparison of diagnostic recommendations from individual physicians versus the collective intelligence of multiple physicians in ambulatory cases referred for specialist consultation Med Decis Making 2022 04 42 3 293 302 10.1177/0272989X211031209 34378444 34378444 PMC8831645 · DOI
  3. [5]Ramnarayan P Tomlinson A Rao A Coren M Winrow A Britto J ISABEL: a web-based differential diagnostic aid for paediatrics: results from an initial performance evaluation Arch Dis Child 2003 05 88 5 408 13 10.1136/adc.88.5.408 12716712 12716712 PMC1719548 · DOI
  4. [6]Müller L Gangadharaiah R Klein SC Perry J Bernstein G Nurkse D Wailes D Graham R El-Kareh R Mehta S Vinterbo SA Aronoff-Spencer E An open access medical knowledge base for community driven diagnostic decision support system development BMC Med Inform Decis Mak 2019 04 27 19 1 93 10.1186/s12911-019-0804-1 31029130 10.1186/s12911-019-0804-1 31029130 PMC6486985 · DOI
  5. [7]Prakash A Zhao S Hasan S Datla V Lee K Qadir A Liu J Farri O Condensed memory networks for clinical diagnostic inferencing Proc AAAI Conf Artif Intell 2017 02 12 31 1 10.1609/aaai.v31i1.10964 · DOI
  6. [9]Liu J Zhang Z Razavian N Deep EHR: chronic disease prediction using medical notes arXiv Preprint posted online August 15, 2018
  7. [10]Gehrmann S Dernoncourt F Li Y Carlson ET Wu JT Welt J Foote J Jr Moseley ET Grant DW Tyler PD Celi LA Comparing deep learning and concept extraction based methods for patient phenotyping from clinical narratives PLoS One 2018 2 15 13 2 e0192360 10.1371/journal.pone.0192360 29447188 PONE-D-17-23063 29447188 PMC5813927 · DOI
  8. [11]Jain D Singh V Feature selection and classification systems for chronic disease prediction: a review Egypt Inform J 2018 11 19 3 179 89 10.1016/j.eij.2018.03.002 · DOI
  9. [12]Singh Kohli P Arora S Application of machine learning in disease prediction Proceedings of the 4th International Conference on Computing Communication and Automation 2018 ICCCA 2018 December 14-15, 2018 Greater Noida, India 10.1109/ccaa.2018.8777449 · DOI
  10. [13]Abdullahi T Nitschke G Sweijd N Predicting diarrhoea outbreaks with climate change PLoS One 2022 4 19 17 4 e0262008 10.1371/journal.pone.0262008 35439258 PONE-D-21-07608 35439258 PMC9017952 · DOI
  11. [14]Alon G Chen E Savova G Eickhoff C Diagnosis prevalence vs. efficacy in machine-learning based diagnostic decision support arXiv Preprint posted online June 24, 2020
  12. [15]Rios A Kavuluru R Few-shot and zero-shot multi-label learning for structured label spaces Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing 2018 EMNLP 2018 October 31-November 4, 2018 Brussels, Belgium 10.18653/v1/d18-1352 PMC6375489 30775726 · DOI
  13. [16]Zhao Y Wong ZS Tsui KL A framework of rebalancing imbalanced healthcare data for rare events' classification: a case of look-alike sound-alike mix-up incident detection J Healthc Eng 2018 05 22 2018 6275435 10.1155/2018/6275435 10.1155/2018/6275435 29951182 29951182 PMC5987310 · DOI
  14. [17]Romera-Paredes B Torr PH Feris R Lampert C Parikh D An embarrassingly simple approach to zero-shot learning Visual Attributes 2017 03 22 Cham, Switzerland Springer
  15. [18]Veeranna SP Nam J Mencía EL Furnkranz J Using semantic similarity for multi-label zero-shot classification of text documents Proceedings of the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning 2016 ESANN 2016 April 27-29, 2016 Bruges, Belgium
  16. [19]Brown TB Mann B Ryder N Subbiah M Kaplan J Dhariwal P Neelakantan A Shyam P Sastry G Askell A Agarwal S Herbert-Voss A Krueger G Henighan T Child R Ramesh A Ziegler DM Wu J Winter C Hesse C Chen M Sigler E Litwin M Gray S Chess B Clark J Berner C McCandlish S Radford A Sutskever I Amodei D Language models are few-shot learners arXiv Preprint posted online May 28, 2020
  17. [20]Sanh V Webson A Raffel C Bach SH Sutawika L Alyafeai Z Chaffin A Stiegler A Scao TL Raja A Dey M Bari MS Xu C Thakker U Sharma S Szczechla E Kim T Chhablani G Nayak N Datta D Chang J Jiang MT Wang H Manica M Shen S Yong ZX Pandey H Bawden R Wang T Neeraj T Rozen J Sharma A Santilli A Fevry T Fries JA Teehan R Bers T Biderman S Gao L Wolf T Rush AM Multitask prompted training enables zero-shot task generalization arXiv Preprint posted online October 15, 2021
  18. [21]Li Y Chao X Semi-supervised few-shot learning approach for plant diseases recognition Plant Methods 2021 06 27 17 1 68 10.1186/s13007-021-00770-1 34176505 10.1186/s13007-021-00770-1 34176505 PMC8237441 · DOI
  19. [22]Zhao Y Lai H Yin J Zhang Y Yang S Jia Z Ma J Zero-shot medical image retrieval for emerging infectious diseases based on meta-transfer learning - worldwide, 2020 China CDC Wkly 2020 12 25 2 52 1004 8 10.46234/ccdcw2020.268 34594825 ccdcw-2-52-1004 34594825 PMC8422228 · DOI
  20. [23]Soldaini L Cohan A Yates A Goharian N Frieder O Retrieving medical literature for clinical decision support Proceedings of the 37th European Conference on IR Research 2015 ECIR 2015 March 29-April 2, 2015 Vienna, Austria 10.1007/978-3-319-16354-3_59 · DOI
  21. [24]Hasan SA Ling Y Liu J Farri O Using neural embeddings for diagnostic inferencing in clinical question answering Proceedings of the Twenty-Fourth Text REtrieval Conference 2015 TREC 2015 November 17-20, 2015 Gaithersburg, MD
  22. [25]Hasan SA Zhu X Dong Y Liu J Farri O A hybrid approach to clinical question answering Proceedings of the Twenty-Third Text REtrieval Conference 2014 2014 TREC 2014 November 19-21, 2014 Gaithersburg, MD
  23. [26]Text REtrieval Conference (TREC) home page Text REtrieval Conference (TREC) 2024-06-03 https://trec.nist.gov/
  24. [27]Naik A Parasa S Feldman S Wang LL Hope T Literature-augmented clinical outcome prediction arXiv Preprint posted online November 16, 2021 10.18653/v1/2022.findings-naacl.33 · DOI
  25. [28]Bodenreider O The Unified Medical Language System (UMLS): integrating biomedical terminology Nucleic Acids Res 2004 01 01 32 Database issue D267 70 10.1093/nar/gkh061 14681409 32/suppl_1/D267 14681409 PMC308795 · DOI · Paper page
  26. [29]White J PubMed 2.0 Med Ref Serv Q 2020 10 21 39 4 382 7 10.1080/02763869.2020.1826228 33085945 33085945 · DOI
  27. [30]Eickhoff C Gmehlin F Patel AV Boullier J Fraser H DC3 -- a diagnostic case challenge collection for clinical decision support Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval 2019 ICTIR '19 October 2-5, 2019 Santa Clara, CA 10.1145/3341981.3344239 · DOI
  28. [31]Soldaini L Goharian N QuickUMLS: a fast, unsupervised approach for medical concept extraction Proceedings of the 2nd SIGIR Workshop on Medical Information Retrieval (MedIR) 2016 SIGIR 2016 July 21, 2016 Pisa, Italy
  29. [32]Zhao Z Jin Q Chen F Peng T Yu S A large-scale dataset of patient summaries for retrieval-based clinical decision support systems Sci Data 2023 12 18 10 1 909 10.1038/s41597-023-02814-8 10.1038/s41597-023-02814-8 38110415 10.1038/s41597-023-02814-8 38110415 PMC10728216 · DOI · Paper page
  30. [33]Vardell E Moore M Isabel, a clinical decision support system Med Ref Serv Q 2011 04 25 30 2 158 66 10.1080/02763869.2011.562800 21534115 936902061 21534115 · DOI
  31. [35]Robertson S Walker S Hancock-Beaulieu MM Gatford M Payne A Okapi at TREC-4 Proceedings of the Fourth Text REtrieval Conference 1995 TREC-4 1995 November 1-3, 1995 Gaithersburg, MD
  32. [36]Jin Q Kim W Chen Q Comeau DC Yeganova L Wilbur WJ Lu Z MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval Bioinformatics 2023 11 01 39 11 2023 10.1093/bioinformatics/btad651 37930897 7335842 PMC10627406 37930897 · DOI · Paper page
  33. [37]Alsentzer E Murphy J Boag W Weng WH Jindi D Naumann T McDermott M Publicly available clinical BERT embeddings Proceedings of the 2nd Clinical Natural Language Processing Workshop 2019 ClinicalNLP 2019 June 7, 2019 Minneapolis, MN 10.18653/v1/w19-1909 · DOI · Paper page
  34. [38]Lee J Yoon W Kim S Kim D Kim S So CH Kang J BioBERT: a pre-trained biomedical language representation model for biomedical text mining Bioinformatics 2020 02 15 36 4 1234 40 10.1093/bioinformatics/btz682 31501885 5566506 31501885 PMC7703786 · DOI · Paper page
  35. [39]Johnson AE Pollard TJ Shen LW Lehman LH Feng M Ghassemi M Moody B Szolovits P Celi LA Mark RG MIMIC-III, a freely accessible critical care database Sci Data 2016 05 24 3 1 160035 10.1038/sdata.2016.35 10.1038/sdata.2016.35 27219127 sdata201635 27219127 PMC4878278 · DOI · Paper page
  36. [40]Gu Y Tinn R Cheng H Lucas M Usuyama N Liu X Naumann T Gao J Poon H Domain-specific language model pretraining for biomedical natural language processing ACM Trans Comput Healthcare 2021 10 15 3 1 1 23 10.1145/3458754 · DOI · Paper page
  37. [42]Liu F Shareghi E Meng Z Basaldella M Collier N Self-alignment pretraining for biomedical entity representations arXiv Preprint posted online October 22, 2020 10.18653/v1/2021.naacl-main.334 · DOI
  38. [43]Yuan Z Zhao Z Sun H Li J Wang F Yu S CODER: knowledge-infused cross-lingual medical term embedding for term normalization J Biomed Inform 2022 02 126 103983 10.1016/j.jbi.2021.103983 34990838 S1532-0464(21)00312-9 34990838 · DOI
  39. [44]Reimers N Gurevych I Sentence-BERT: sentence embeddings using Siamese BERT-networks arXiv Preprint posted online August 27, 2019 10.18653/v1/d19-1410 · DOI · Paper page
  40. [45]Cormack GV Clarke CL Buettcher S Reciprocal rank fusion outperforms condorcet and individual rank learning methods Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval 2009 SIGIR '09 July 19-23, 2009 Boston, MA 10.1145/1571941.1572114 · DOI · Paper page
  41. [46]Kurland O Culpepper J Fusion in information retrieval: SIGIR 2018 half-day tutorial Proceedings of the 41st International ACM SIGIR Conference on Research & Development in Information Retrieval 2018 SIGIR '18 July 8-12, 2018 Ann Arbor, MI 10.1145/3209978.3210186 · DOI
  42. [47]Mugisha C Paik I Comparison of neural language modeling pipelines for outcome prediction from unstructured medical text notes IEEE Access 2022 10 16489 98 10.1109/access.2022.3148279 · DOI
  43. [48]Yao L Jin Z Mao C Zhang Y Luo Y Traditional Chinese medicine clinical records classification with BERT and domain specific corpora J Am Med Inform Assoc 2019 12 01 26 12 1632 6 10.1093/jamia/ocz164 31550356 5573314 31550356 PMC7647141 · DOI
  44. [49]Shen Z Schutte D Yi Y Bompelli A Yu F Wang Y Zhang R Classifying the lifestyle status for Alzheimer's disease from clinical notes using deep learning with weak supervision BMC Med Inform Decis Mak 2022 07 07 22 Suppl 1 88 10.1186/s12911-022-01819-4 35799294 10.1186/s12911-022-01819-4 35799294 PMC9261217 · DOI
  45. [50]Craswell N Liu L Özsu MT Mean reciprocal rank Encyclopedia of Database Systems 2009 Boston, MA Springer
  46. [52]Abdullahi T Singh R Eickhoff C Learning to make rare and complex diagnoses with generative AI assistance: qualitative study of popular large language models JMIR Med Educ 2024 02 13 10 e51391 10.2196/51391 38349725 v10i1e51391 38349725 PMC10900078 · DOI · Paper page
  47. [53]McGauran N Wieseler B Kreis J Schüler YB Kölsch H Kaiser T Reporting bias in medical research - a narrative review Trials 2010 04 13 11 1 37 10.1186/1745-6215-11-37 20388211 1745-6215-11-37 20388211 PMC2867979 · DOI · Paper page
  48. [54]Montori VM Smieja M Guyatt GH Publication bias: a brief review for clinicians Mayo Clin Proc 2000 12 75 12 1284 8 10.4065/75.12.1284 11126838 S0025-6196(11)63030-9 11126838 · DOI

Cited by

Sorted by citations

Authors
Abdullahi, T., Mercurio, L., Singh, R., & Eickhoff, C.
Affiliations
  • Brown University (US)
  • University of Tübingen (DE)
Corresponding author
Eickhoff, Carsten
ORCID
Venue
JMIR Medical Informatics
Publisher
JMIR Publications Inc.
Published
2024-06-19
Volume
12
Pages
e50209
ISSN
2291-9694 (ISSN-L)
DOI
10.2196/50209
PMC
PMC11222760
OpenAlex
W4394886691
Cited by
20
References
4
Citations by year
2026: 8 · 2025: 10 · 2024: 1 · 2023: 1
FWCI
2.80
Type
journal article
Language
en
Open access
Licence
cc-by
Locations
  • JMIR Medical Informatics (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
  • Eberhard Karls Universität Tübingen
  • National Institutes of Health
  • National Institute on Drug Abuse (T32DA013911)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Physical Sciences › Computer Science › Information Systems › Information Retrieval and Search Behavior
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
Keywords
diagnostic decision support, information retrieval, data sparsity, zero-shot learning, clinical text classification, ClinicalBERT, ensemble learning, diagnostic errors, MIMIC-III, PubMed abstracts, Metathesaurus

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review