CoChat
Sign Up
Cited by 64Biomedical Text Mining and OntologiesTopic ModelingMachine Learning in Healthcare

Biolord-2023: Semantic textual representations fusing large language models and clinical knowledge graph insights

Open access
Remy, François, Demuynck, Kris, Demeester, Thomas·2024·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocae029PMC PMC11339519
Open this paper in CoChat — free accountOpen PDF

Abstract

Objective: In this study, we investigate the potential of large language models (LLMs) to complement biomedical knowledge graphs in the training of semantic models for the biomedical and clinical domains. Materials and Methods: Drawing on the wealth of the Unified Medical Language System knowledge graph and harnessing cutting-edge LLMs, we propose a new state-of-the-art approach for obtaining high-fidelity representations of biomedical concepts and sentences, consisting of 3 steps: an improved contrastive learning phase, a novel self-distillation phase, and a weight averaging phase. Results: Through rigorous evaluations of diverse downstream tasks, we demonstrate consistent and substantial improvements over the previous state of the art for semantic textual similarity (STS), biomedical concept representation (BCR), and clinically named entity linking, across 15+ datasets. Besides our new state-of-the-art biomedical model for English, we also distill and release a multilingual model compatible with 50+ languages and finetuned on 7 European languages. Discussion: Many clinical pipelines can benefit from our latest models. Our new multilingual model enables a range of languages to benefit from our advancements in biomedical semantic representation learning, opening a new avenue for bioinformatics researchers around the world. As a result, we hope to see BioLORD-2023 becoming a precious tool for future biomedical applications. Conclusion: In this article, we introduced BioLORD-2023, a state-of-the-art model for STS and BCR designed for the clinical domain.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocae029

Cites

4 of 4 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Using o4-mini as an LLM judge and BioLORD as a vendor-neutral retrieval-based judge reduces self-preference bias.

    “Our primary evaluation uses o4-mini as an LLM judge, scoring freetext differential diagnoses against reference labels.To reduce potential self-preference bias, we also implement a vendor-neutral, retrieval-based judge: BioLORD (FRemyCompany/BioLORD-2023) (Remy et al., 2024).”

    From Do Mixed-Vendor Multi-Agent LLMs Improve Clinical Diagnosis? (2026) · cites this paper as Remy et al., 2024

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 56

  1. [1]Houssein EH, Rehab EM, Abdelmgeid AA. Machine learning techniques for biomedical natural language processing: a comprehensive review. IEEE Access. 2021; 9: 140628- 140653.
  2. [2]Shi J, Yuan Z, Guo W, Ma C, Chen J, Zhang M. Knowledge-graph-enabled biomedical entity linking: a survey. World Wide Web. 2023; 26(5): 2593- 2622.
  3. [3]Pan S, Luo L, Wang Y, Chen C, Wang J, Wu X. Unifying large language models and knowledge graphs: a roadmap. arXiv. June 14, 2023, preprint: not peer reviewed.
  4. [4]Satvik G, Shivam P, Somya G. Navigating healthcare insights: a birds eye view of explainability with knowledge graphs. In: 2023 IEEE Sixth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE). 2023: 54- 61.
  5. [5]Lin X, Dai L, Zhou Y, et al Comprehensive evaluation of deep and graph learning on drug-drug interactions prediction. Brief Bioinform. 2023; 24(4): 1- 20. 37401373
  6. [6]Wang B, Xie Q, Pei J, et al Pre-trained language models in biomedical domain: a systematic survey. ACM Comput Surv. 2023; 56(3): 1- 52.
  7. [7]Hu L, Liu Z, Zhao Z, Hou L, Nie L, Li J. A survey of knowledge enhanced pre-trained language models. IEEE Tran Knowl Data Eng. 2023. 10.1109/TKDE.2023.3310002 · DOI
  8. [8]Feng Z, Ma W, Yu W, et al Trends in integration of knowledge and large language models: a survey and taxonomy of methods, benchmarks, and applications. arXiv. November 10, 2023, preprint: not peer reviewed.
  9. [9]Sung M, Jeon H, Lee J, Kang J. Biomedical entity representations with synonym marginalization. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, July 2020: 3641- 3650.
  10. [10]Liu F, Shareghi E, Meng Z, Basaldella M, Collier N. Self-alignment pretraining for biomedical entity representations. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2021: 4228- 4238.
  11. [11]Remy F, Demuynck K, Demeester T. BioLORD: learning ontological representations from definitions for biomedical concepts and their textual descriptions. In: Findings of the Association for Computational Linguistics: EMNLP 2022, December 2022: 1454- 1465.
  12. [12]Devlin J, Chang M-W, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019: 4171- 4186.
  13. [13]Lewis P, Perez E, Piktus A, et al Retrieval-augmented generation for knowledge-intensive NLP tasks. In: Proceedings of the 34th International Conference on Neural Information Processing Systems, December 2020: 9459- 9474.
  14. [14]Kim J, Choi S, Amplayo RK, Hwang SW. Retrieval-augmented controllable review generation. In: Proceedings of the 28th International Conference on Computational Linguistics, December 2020: 2284- 2295.
  15. [16]Bodenreider O. The Unified Medical Language System (UMLS): integrating biomedical terminology. Nucleic Acids Res. 2004; 32(Database issue): D267- D270. 14681409 PMC308795
  16. [17]Coté RA, Robboy S. Progress in medical information management. Systematized nomenclature of medicine (SNOMED). JAMA. 1980; 243(8): 756- 762. 6986000
  17. [18]Remy F, Demuynck K, Demeester T. Automatic Glossary of Clinical Terminology: a large-scale dictionary of biomedical definitions generated from ontological knowledge. In: The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks, July 2023: 265- 272.
  18. [19]Reimers N, Gurevych I. Making monolingual sentence embeddings multilingual using knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, November 2020: 4512- 4525.
  19. [20]Feng F, Yang Y, Cer D, Arivazhagan N, Wang W. Language-agnostic BERT sentence embedding. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, May 2022: 878- 891.
  20. [21]Liu F, Vulić I, Korhonen A, Collier N. Learning domain-specialised representations for cross-lingual biomedical entity linking. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, vol. 2, August 2021: 565- 574.
  21. [22]Cui H, Lu J, Wang S, et al A survey on knowledge graphs for healthcare: resources, applications, and promises. arXiv. August 23, 2023, preprint: not peer reviewed. PMC12995551 40712809
  22. [23]Yang X, Chen A, PourNejatian N, et al A large language model for electronic health records. NPJ Digit Med. 2022; 5(1): 194. 36572766 PMC9792464
  23. [26]Venkatesh KP, Raza MM, Kvedar JC. Automating the overburdened clinical coding system: challenges and next steps. NPJ Digit Med. 2023; 36(1): 16. PMC9898522 36737496
  24. [27]Wu C, Lei J, Zheng Q, et al Can GPT-4V(ision) serve medical applications? Case studies on GPT-4V for multimodal medical diagnosis. arXiv. October 17, 2023, preprint: not peer reviewed.
  25. [28]Yan Z, Zhang K, Zhou R, He L, Li X, Sun L. Multimodal ChatGPT for medical applications: an experimental study of GPT-4V. arXiv. October 29, 2023, preprint: not peer reviewed.
  26. [29]Jin Q, Wang Z, Floudas CS, Sun J, Lu Z. Matching patients to clinical trials with large language models. arXiv. July 28, 2023, preprint: not peer reviewed. PMC11574183 39557832
  27. [30]Taylor R, Kardas M, Cucurull G, et al Galactica: a large language model for science. arXiv. November 16, 2022, preprint: not peer reviewed.
  28. [31]Wang G, Yang G, Du Z, Fan L, Li X. ClinicalGPT: large language models finetuned with diverse medical data and comprehensive evaluation. arXiv. June 16, 2023, preprint: not peer reviewed.
  29. [33]Luo R, Sun L, Xia Y, et al BioGPT: generative pre-trained transformer for biomedical text generation and mining. Brief Bioinform. 2022; 23(6): bbac409. 36156661
  30. [34]OpenAI. GPT-4 technical report. arXiv. March 15, 2023, preprint: not peer reviewed.
  31. [35]Oord AV, Li Y, Vinyals O. Representation learning with contrastive predictive coding. arXiv. July 10, 2018, preprint: not peer reviewed.
  32. [36]Remy F, Scaboro S, Portelli B. Boosting adverse drug event normalization on social media: general-purpose model initialization and biomedical semantic text similarity benefit zero-shot linking in informal contexts. In: Proceedings of the Eleventh International Workshop on Natural Language Processing for Social Media, November 1, 2023: 47- 52.
  33. [37]Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, November 2019: 3982- 3992.
  34. [38]Ribeiro J, Melo FS, Dias J. Multi-task learning and catastrophic forgetting in continual reinforcement learning. EPiC Ser Comput. 2019; 65: 163- 175.
  35. [39]He T, Liu J, Cho K, et al Analyzing the forgetting problem in pretrain-finetuning of open-domain dialogue response models. In: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, May 2021: 1121- 1133.
  36. [40]Gao T, Yao X, Chen D. SimCSE: simple contrastive learning of sentence embeddings. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, November 2021: 6894- 6910.
  37. [41]Li B, Zhou H, He J, Wang M, Yang Y, Li L. On the sentence embeddings from pre-trained language models. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, November 2020: 9119- 9130.
  38. [42]Ethayarajh K. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, November 2019: 55- 65.
  39. [43]Wortsman M, Ilharco G, Gadre SY, et al Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In: International Conference on Machine Learning, vol. 162, 2022: 23965- 23998.
  40. [44]Remy F, De Jaeger P, Demuynck K. Taming large lexicons: translating clinical text using medical ontologies and sentence templates. In: EmP: 1st RADar Conference on Engineer Meets Physician, 2022: 1- 5.
  41. [46]Romanov A, Shivade C. Lessons from natural language inference in the clinical domain. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, November 2018, 1586- 1596.
  42. [47]Sogancioglu G, Öztürk H, Özgür A. BIOSSES: a semantic sentence similarity estimation system for the biomedical domain. Bioinformatics. 2017; 33(14): i49- i58. 28881973 PMC5870675
  43. [48]Marelli M, Menini S, Baroni M, Bentivogli L, Bernardi R, Zamparelli R. A SICK cure for the evaluation of compositional distributional semantic models. In: Proceedings of the Ninth International Conference on Language Resources and Evaluation, May 2014: 216- 233.
  44. [49]Cer D, Diab M, Agirre E, Lopez-Gazpio I, Specia L. SemEval-2017 Task 1: semantic textual similarity multilingual and crosslingual focused evaluation. In: Proceedings of the 11th International Workshop on Semantic Evaluation, August 2017: 1- 14.
  45. [50]Ofer D, Linial M. Automated annotation of disease subtypes. medRxiv. September 25, 2023), preprint: not peer reviewed. 38701887
  46. [51]Kalyan KS, Sangeetha S. Hybrid approach to measure semantic relatedness in biomedical concepts. arXiv. January 25, 2021, preprint: not peer reviewed.
  47. [52]Schulz C, Levy-Kramer J, Van Assel C, Kepes M, Hammerla N. Biomedical concept relatedness—a large EHR-based benchmark. In: Proceedings of the 28th International Conference on Computational Linguistics, December 2020: 6565- 6575.
  48. [53]Pakhomov SV, McInnes B, Adam T, Liu Y, Pedersen T, Melton GB. Semantic similarity and relatedness between clinical terms: an experimental study. AMIA Annu Symp. Proc. 2010; 2010: 572. 21347043 PMC3041430
  49. [54]Pakhomov SV, Pedersen T, McInnes B, Melton GB, Ruggieri A, Chute CG. Towards a framework for developing semantic relatedness reference standards. J Biomed Inform. 2011; 44(2): 251- 265. 21044697 PMC3063326
  50. [55]Portelli B, Scaboro S, Santus E, Sedghamiz H, Chersoni E, Serra G. Generalizing over long tail concepts for medical term normalization. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, December 2022: 8580- 8591.
  51. [56]Alvaro N, Miyao Y, Collier N. TwiMed: twitter and PubMed comparable corpus of drugs, diseases, symptoms, and their relations. JMIR Public Health Surveill. 2017; 3(2): e24. 28468748 PMC5438461
  52. [57]Gonzalez-Hernandez G, Klein AZ, Flores I, et al Overview of the Fifth Social Media Mining for Health Applications (#SMM4H) Shared Tasks at COLING 2020. In: Proceedings of the Fifth Social Media Mining for Health Applications Workshop & Shared Task. Association for Computational Linguistics, December 2020: 27- 36.
  53. [58]Zolnoori M, Fung KW, Patrick TB, et al The PsyTAR dataset: From patients generated narratives to a corpus of adverse drug events and effectiveness of psychiatric medications. Data Brief. 2019; 24: 103838. 31065579 PMC6495095
  54. [59]Karimi S, Metke-Jimenez A, Kemp M, Wang C. Cadec: a corpus of adverse drug event annotations. J Biomed Inform. 2015; 55: 73- 81. 25817970
  55. [60]Jin Q, Kim W, Chen Q, et al MedCPT: contrastive pre-trained transformers with large-scale PubMed search logs for zero-shot biomedical information retrieval. Bioinformatics. 2023; 39(11): btad651. 37930897 PMC10627406
  56. [61]Mao Y, Fung KW. Use of word and graph embedding to measure semantic relatedness between Unified Medical Language System concepts. J Am Med Inform Assoc. 2020; 27(10): 1538- 1546. 33029614 PMC7566472

Cited by

Sorted by citations

Authors
Remy, F., Demuynck, K., & Demeester, T.
Affiliations
Ghent University (BE)
Corresponding author
Remy, François
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2024-02-27
Volume
31
Issue
9
Pages
1844-1855
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocae029
PMC
PMC11339519
OpenAlex
W4392186815
Cited by
64
References
4
Citations by year
2026: 27 · 2025: 26 · 2024: 11
FWCI
8.71
Type
journal article
Language
en
Open access
Licence
Locations
  • Journal of the American Medical Informatics Association (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Ghent University Academic Bibliography (Ghent University) (submittedVersion, cc-by)
Funders
  • ADAM
  • VLAIO O&O (HBC.2020.3234)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
Keywords
semantic textual similarity, large language models, biomedical knowledge graphs, UMLS, contrastive learning, self-distillation, multilingual representation learning, clinical text representation
MeSH terms
Humans, Natural Language Processing, Semantics, Unified Medical Language System

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review