Learning a Health Knowledge Graph from Electronic Medical Records
Open accessAbstract
Demand for clinical decision support systems in medicine and self-diagnostic symptom checkers has substantially increased in recent years. Existing platforms rely on knowledge bases manually compiled through a labor-intensive process or automatically derived using simple pairwise statistics. This study explored an automated process to learn high quality knowledge bases linking diseases and symptoms directly from electronic medical records. Medical concepts were extracted from 273,174 de-identified patient records and maximum likelihood estimation of three probabilistic models was used to automatically construct knowledge graphs: logistic regression, naive Bayes classifier and a Bayesian network using noisy OR gates. A graph of disease-symptom relationships was elicited from the learned parameters and the constructed knowledge graphs were evaluated and validated, with permission, against Google’s manually-constructed knowledge graph and against expert physician opinions. Our study shows that direct and automated construction of high quality health knowledge graphs from medical records using rudimentary concept extraction is feasible. The noisy OR model produces a high quality knowledge graph reaching precision of 0.85 for a recall of 0.6 in the clinical evaluation. Noisy OR significantly outperforms all tested models across evaluation frameworks (p < 0.01).
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Histopathological images and EHRs are two modes of patient data successfully integrated using deep graph learning.
“Two modes of patient data successfully integrated using deep graph learning are histopathological images [8, 185] and EHRs [186, 187].”
From Graph representation learning in biomedicine and healthcare (2022) · cites this paper as [187]
Electronic health records are typically represented by ICD codes.
“Electronic health records are typically represented by ICD (International Classification of Disease) codes [186, 187].”
From Graph representation learning in biomedicine and healthcare (2022) · cites this paper as [187]
An ST-GCN is designed to utilize underlying spatial and temporal dependencies of EHR data for generating patient diagnoses.
“An ST-GCN [78] is designed to utilize the underlying spatial and temporal dependencies of EHR data for generating patient diagnoses [187].”
From Graph representation learning in biomedicine and healthcare (2022) · cites this paper as [187]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970 · see citing works
Goldberger, A. L., Amaral, L. A. N., Glass, L., Hausdorff, J. M., Ivanov, P. C., Mark, R. G., Mietus, J. E., Moody, G. B., Peng, C.-K., & Stanley, H. E. (2000) · Circulation · doi:10.1161/01.cir.101.23.e215 · Cited by 14,809 · see citing works
Causality: Models, Reasoning, and Inference
Pearl, J. (2009) · doi:10.1017/cbo9780511803161 · Cited by 11,653 · see citing works
(2012) · Cited by 9,062 · see citing works
A Simple Algorithm for Identifying Negated Findings and Diseases in Discharge Summaries
Chapman, W. W., Bridewell, W. B., Hanbury, P., Cooper, G. F., & Buchanan, B. G. (2001) · Journal of Biomedical Informatics · doi:10.1006/jbin.2001.1029 · Cited by 1,121 · see citing works
Computer-aided Diagnosis of Acute Abdominal Pain
de Dombal, F. T., Leaper, D. J., Staniland, J. R., McCann, A. P., & Horrocks, J. C. (1972) · BMJ · doi:10.1136/bmj.2.5804.9 · Cited by 675 · see citing works
How reliable are the results of large-scale information retrieval experiments?
Zobel, J. (1998) · Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval · doi:10.1145/290941.291014 · Cited by 579 · see citing works
Cyberchondria: studies of the escalation of medical concerns in web search
White, R. W., & Horvitz, E. (2009) · ACM Transactions on Information Systems · doi:10.1145/1629096.1629101 · Cited by 546 · see citing works
Rosenbaum, P. R. (2002) · Springer Series in Statistics · doi:10.1007/978-1-4757-3692-2_1 · Cited by 484 · see citing works
Miller, R. A. (1994) · Journal of the American Medical Informatics Association · doi:10.1136/jamia.1994.95236141 · Cited by 460 · see citing works
Automated de-identification of free-text medical records
Neamatullah, I., Douglass, M. M., Lehman, L.-W. H., Reisner, A., Villarroel, M., Long, W. J., Szolovits, P., Moody, G. B., Mark, R. G., & Clifford, G. D. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-32 · Cited by 413 · see citing works
Ford, E. M., Carroll, J. A., Smith, H. E., Scott, D., & Cassell, J. (2016) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocv180 · Cited by 411 · see citing works
DXplain. An evolving diagnostic decision-support system
Barnett, G. O. (1987) · JAMA · doi:10.1001/jama.258.1.67 · Cited by 391 · see citing works
Googling for a diagnosis—use of Google as a diagnostic aid: internet based study
Tang, H., & Ng, J. (2006) · BMJ · doi:10.1136/bmj.39003.640567.ae · Cited by 386 · see citing works
Learning Bayesian network parameters from small data sets: application of Noisy-OR gates
Oniśko, A., Drużdżel, M. J., & Wasyluk, H. (2001) · International Journal of Approximate Reasoning · doi:10.1016/s0888-613x(01)00039-1 · Cited by 331 · see citing works
Shwe, M., Blackford Middleton, A., Heckerman, D. E., Henrion, M., Horvitz, E., Lehmann, H. P., & Cooper, G. F. (1991) · PubMed · Cited by 293 · see citing works
Barnett, G. O. (1987) · JAMA · doi:10.1001/jama.1987.03400010071030 · Cited by 285 · see citing works
Yearbook of Medical Informatics
(2013) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1638865 · Cited by 237 · see citing works
Population-Level Prediction of Type 2 Diabetes From Claims Data and Analysis of Risk Factors
Razavian, N., Blecker, S., Schmidt, A. M., Smith–McLallen, A., Nigam, S., & Sontag, D. A. (2015) · Big Data · doi:10.1089/big.2015.0020 · Cited by 212 · see citing works
Self-citationProbabilistic Diagnosis Using a Reformulation of the INTERNIST-1/QMR Knowledge Base
Shwe, M., Heckerman, D. E., Henrion, M., Horvitz, E., Lehmann, H. P., Cooper, G. F., & Blackford Middleton, A. (1991) · Methods of Information in Medicine · doi:10.1055/s-0038-1634846 · Cited by 173 · see citing works
MYCIN: a knowledge-based consultation program for infectious disease diagnosis
van Melle, W. (1978) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(78)80049-2 · Cited by 148 · see citing works
The sensitivity of belief networks to imprecise probabilities: an experimental investigation
Pradhan, M., Henrion, M., Provan, G. M., et al. (1996) · Artificial Intelligence · doi:10.1016/0004-3702(96)00002-1 · Cited by 134 · see citing works
Paparrizos, J., White, R. W., & Horvitz, E. (2016) · Journal of Oncology Practice · doi:10.1200/jop.2015.010504 · Cited by 132 · see citing works
Sick patients have more data: the non-random completeness of electronic health records.
Weiskopf, N. G., Rusanov, A., & Weng, C. (2013) · Europe PMC (PubMed Central) · Cited by 99 · see citing works
Building the graph of medicine from millions of clinical narratives
Finlayson, S. G., LePendu, P., & Shah, N. H. (2014) · Scientific Data · doi:10.1038/sdata.2014.32 · Cited by 91 · see citing works
WatsonPaths: Scenario‐Based Question Answering and Inference over Unstructured Information
Lally, A., Bagchi, S., Barborak, M., Buchanan, D. W., Chu‐Carroll, J., Ferrucci, D. A., Glass, M. R., Kalyanpur, A. A., Mueller, E. T., Murdock, J. W., et al. (2017) · AI Magazine · doi:10.1609/aimag.v38i2.2715 · Cited by 87 · see citing works
Integration of tools for binding archetypes to SNOMED CT
Sundvall, E., Qamar, R., Nyström, M., et al. (2008) · BMC Medical Informatics and Decision Making · doi:10.1186/1472-6947-8-s1-s7 · Cited by 75 · see citing works
The information-seeking behavior of clinical staff in a large health care organization
Hider, P. N., Griffin, G., Walker, M., et al. (2009) · Journal of the Medical Library Association JMLA · doi:10.3163/1536-5050.97.1.009 · Cited by 73 · see citing works
ILIAD as an Expert Consultant to Teach Differential Diagnosis
Warner, H. R., Haug, P. J., Bouhaddou, O., Lincoln, M. J., Sorenson, D. K., Williamson, J., & Fan, C. (1988) · PubMed Central · Cited by 73 · see citing works
Use of the Quick Medical Reference (QMR) Program as a Tool for Medical Education
Miller, R. A., & Masarie, F. E. (1989) · Methods of Information in Medicine · doi:10.1055/s-0038-1636814 · Cited by 65 · see citing works
Deas, P. M. (1877) · BMJ · doi:10.1136/bmj.2.862.9 · Cited by 60 · see citing works
Goodwin, T. R., & Harabagiu, S. M. (2013) · IEEE International Conference on Semantic Computing (ICSC) · doi:10.1109/icsc.2013.68 · Cited by 48 · see citing works
Giving Patients Choice and Control: Health Informatics on the Patient Journey
Gann, B. (2012) · Yearbook of Medical Informatics · doi:10.1055/s-0038-1639433 · Cited by 38 · see citing works
Sondhi, P., Sun, J., Tong, H., & Zhai, C. (2012) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/2339530.2339712 · Cited by 33 · see citing works
Accuracy of a Computer-Based Diagnostic Program for Ambulatory Patients With Knee Pain
Bisson, L. J., Komm, J. T., Bernas, G. A., Fineberg, M. S., Marzo, J. M., Rauh, M. A., Smolinski, R. J., & Wind, W. M. (2014) · The American Journal of Sports Medicine · doi:10.1177/0363546514541654 · Cited by 29 · see citing works
Probabilistic asthma case finding: a noisy or reformulation.
Anand, V., & Downs, S. M. (2008) · PubMed · Cited by 21 · see citing works
ISABEL: a novel Internet-delivered clinical decision support system
Ramnarayan, P., Kulkarni, G., Tomlinson, A., et al. (2004) · Cited by 13 · see citing works
(2016) · Children and Young People Now · doi:10.12968/cypn.2016.20.28 · Cited by 9 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- [17]Search for medical information on Google. Vol. 2016 (Search Help, 2016).
- [18]Pinchin, V. I’m Feeling Yucky: Searching for symptoms on Google. Vol. 2016 (Google Official Blog, 2016).
- [23]Sondhi, P., Sun, J., Tong, H. & Zhai, C. SympGraph: a framework for mining clinical notes through symptom relation graphs. In Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining 1167–1175 (ACM, 2012).
- [28]Jernite, Y., Halpern, Y., Horng, S. & Sontag, D. Predicting chief complaints at triage time in the emergency department. in NIPS 2013 Workshop on Machine Learning for Clinical Data Analysis and Healthcare (2013).
Cited by
Sorted by citations
Graph representation learning in biomedicine and healthcare
Li, M. M., Huang, K., & Zitnik, M. (2022) · Nature Biomedical Engineering · doi:10.1038/s41551-022-00942-x · Cited by 294 · see citing works
“Methods that embed medical entities, including EHRs and medical ontologies, leverage the inherently hierarchical structure of knowledge graphs of medical concepts 187 .”
Medical large language models are vulnerable to data-poisoning attacks
Alber, D. A., Yang, Z., Alyakin, A., Yang, E., Rai, S., Valliani, A. A., Zhang, J., Rosenbaum, G. R., Amend-Thomas, A. K., Kurland, D. B., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03445-1 · Cited by 195 · see citing works
“Centralized organization or computer-aided approaches may ameliorate some maintenance issues, and bespoke knowledge graphs compiled from electronic health records 59 raise the possibility of tailoring our defensive technique to institutions.”
Machine learning for biochemical engineering: A review
Mowbray, M., Savage, T., Wu, C., Song, Z., Cho, B. A., Del Rio-Chanona, E. A., & Zhang, D. (2021) · Biochemical Engineering Journal · doi:10.1016/j.bej.2021.108054 · Cited by 188 · see citing works
Automated clinical knowledge graph generation framework for evidence based medicine.
Alam, F., Giglou, H. B., & Malik, K. M. (2023) · Expert Systems with Applications · doi:10.1016/j.eswa.2023.120964 · Cited by 27 · see citing works
A Method to Learn Embedding of a Probabilistic Medical Knowledge Graph: Algorithm Development
Li, L., Wang, P., Wang, Y., Wang, S., Yan, J., Jiang, J., Tang, B., Wang, C., & Liu, Y. (2020) · JMIR Medical Informatics · doi:10.2196/17645 · Cited by 25 · see citing works
Unlocking electronic health records: a hybrid graph RAG approach to safe clinical AI for patient QA
Thio, S., Lewis, M., Denaxas, S., & Dobson, R. J. B. (2026) · Frontiers in Digital Health · doi:10.3389/fdgth.2026.1780700 · Cited by 6 · see citing works
- Authors
- Rotmensch, M., Halpern, Y., Tlimat, A., Horng, S., & Sontag, D.
- Affiliations
- New York University (US)
- Beth Israel Deaconess Medical Center (US)
- MIT Computer Science and Artificial Intelligence Laboratory (US)
- Corresponding author
- Sontag, David
- Venue
- Scientific Reports
- Published
- 2017-07-20
- Volume
- 7
- Issue
- 1
- Pages
- 5994-5994
- ISSN
- 2045-2322 (ISSN-L)
- PMC
- PMC5519723
- OpenAlex
- W2735585131
- Cited by
- 423
- References
- 38
- Citations by year
- 2026: 38 · 2025: 40 · 2024: 51 · 2023: 66 · 2022: 51 · 2021: 69 · 2020: 55 · 2019: 34 · 2018: 17 · 2017: 2
- FWCI
- 12.07
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Scientific Reports (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
- DSpace@MIT (Massachusetts Institute of Technology) (submittedVersion, cc-by-nc)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Biomedical Text Mining and Ontologies
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Keywords
- electronic health records, clinical decision support, symptom checkers, noisy-OR model, Bayesian networks, medical concept extraction, probabilistic models, clinical validation
- MeSH terms
- Adult, Disease, Humans, Middle Aged, Knowledge Bases, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
