CoChat
Sign Up
Cited by 123Machine Learning in HealthcareArtificial Intelligence in HealthcareLipoproteins and Cardiovascular Health

Finding missed cases of familial hypercholesterolemia in health systems using machine learning

Open access
Banda, Juan M., Sarraju, Ashish, Abbasi, Fahim, Parizo, Justin, Pariani, Mitchel, Ison, Hannah, Briskin, Elinor, Wand, Hannah +10 more·2019·npj Digital Medicine
DOI 10.1038/s41746-019-0101-5PMC PMC6550268
Open this paper in CoChat — free accountOpen PDF

Abstract

Familial hypercholesterolemia (FH) is an underdiagnosed dominant genetic condition affecting approximately 0.4% of the population and has up to a 20-fold increased risk of coronary artery disease if untreated. Simple screening strategies have false positive rates greater than 95%. As part of the FH Foundation′s FIND FH initiative, we developed a classifier to identify potential FH patients using electronic health record (EHR) data at Stanford Health Care. We trained a random forest classifier using data from known patients ( n = 197) and matched non-cases ( n = 6590). Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 on a held-out test-set. We evaluated the accuracy of the classifier′s predictions by chart review of 100 patients at risk of FH not included in the original dataset. The classifier correctly flagged 84% of patients at the highest probability threshold, with decreasing performance as the threshold lowers. In external validation on 466 FH patients (236 with genetically proven FH) and 5000 matched non-cases from the Geisinger Healthcare System our FH classifier achieved a PPV of 0.85. Our EHR-derived FH classifier is effective in finding candidate patients for further FH screening. Such machine learning guided strategies can lead to effective identification of the highest risk patients for enhanced management strategies.

AI snapshot

Written from the paper’s full text.

TL;DR

A random forest classifier using structured and unstructured EHR data from Stanford identified patients likely to have familial hypercholesterolemia, with internal PPV 0.88 and sensitivity 0.75 and external Geisinger PPV 0.85. Chart review of high-probability flags found 84% met DLCN ≥3 or MEDPED criteria, suggesting the model can prioritize undiagnosed FH cases for further evaluation.

Key findings

  • On a held-out Stanford test set, the classifier achieved PPV 0.88, sensitivity 0.75, specificity 0.99, F1 0.81, AUROC 0.94, and AUPRC 0.71.
  • External validation at Geisinger achieved PPV 0.85, sensitivity 0.68, specificity 0.99, F1 0.75, AUROC 0.94, and AUPRC 0.68; limiting to genetically confirmed cases gave an F1 score of 0.82.
  • In chart review of 100 held-out patients, among 56 with classifier probability 0.99–0.90, 47/56 (84%) had DLCN ≥3 or were MEDPED positive; only 3 of the flagged chart-review-confirmed likely FH patients had been clinically diagnosed with FH.
  • Top classifier features included LDL cholesterol text mentions, high/very high cholesterol and LDL lab bins, cardiology clinic mentions, atorvastatin/ezetimibe/rosuvastatin prescriptions, and hyperlipidemia diagnosis codes.

Methods

  • A random forest classifier was built using OMOP-CDM/APHRODITE with structured (labs, diagnoses, drugs) and unstructured (clinical notes) EHR data; features were normalized by follow-up length, rare features removed, and lab values binned into five categories.
  • Stanford training used 197 known FH patients and matched non-cases (n = 6,590 total: 5,400 training controls and 1,190 test controls); cases were split into 180 training and 17 test, with 1:70 test prevalence, 10-fold cross-validation, and repeated train/test splits; chart review of 100 held-out flagged patients used DLCN/MEDPED criteria.
  • External validation used Geisinger data with 466 FH cases (236 genetically confirmed) and 5,000 matched non-cases; 71 cases and 4,970 controls were sampled at 1:70 prevalence and evaluated over 10 iterations.

Limitations

  • Stated: the Stanford test set assumed 1:70 FH prevalence, so performance may differ if real lipid-clinic prevalence differs; downstream outcomes such as diagnosis rates, cascade screening, ASCVD events, or survival were not assessed; the classifier was not trained on pooled data because that would prevent external-validity assessment; and the positive sample was relatively small (n = 197 at Stanford).
  • Stated: cost-effectiveness discussion assumes follow-up costs for chart review and genetic testing, and the authors prioritized PPV over recall because those follow-up actions are expensive.
  • Evident from the text: training was retrospective and single-site, while chart review and external validation used sampled case-control ratios rather than full native ratios, so estimates depend on the chosen prevalence; top features include non-traditional variables (e.g., paroxysmal supraventricular tachycardia, “Cytologic”) whose clinical relevance or confounding is not resolved.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1038/s41746-019-0101-5

Cites

38 of 38 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Claims the paper makes

22 of 36 cited sources checked · 10 supported · 0 flagged · accuracy not yet measured

  • Premisecitation

    Less than 10% of persons with familial hypercholesterolemia in the United States have been diagnosed.

    “it is estimated that less than 10% of persons with FH in the US have been diagnosed,1”
    • Also stated in: § Discussion · hedged
      “It is estimated that only ~10% of patients with FH are diagnosed in the United States.7”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source

      The introductory section says FH remains underdiagnosed worldwide, which supports the general point. The Diagnosis and Screening sections discuss barriers to recognition, but the passages do not give a US estimate or say that fewer than 10% have been diagnosed. Page numbers were not supplied. (LLM, conf=0.83)

      Experimental — accuracy not yet measured
    • cites [7] Knowles, J. W., Rader, D. J. & Khoury, M. J. Cascade screening for familial hypercholesterolemia and the use of genetic testing. JAMA 318, 381–382 (2017). source
  • Premisecitation

    In adults with LDL-C greater than 190 mg/dl, less than 5% will be found to harbor a causal familial hypercholesterolemia gene mutation.

    “While high LDL-C is a cardinal feature of FH, less than 5% of adults with an LDL-C > 190 mg/dl will be found to harbor a causal FH gene mutation.3”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). source
  • Premisecitation

    Cascade screening of relatives after identifying an individual with familial hypercholesterolemia is highly cost-effective in reducing excess morbidity in family members.

    “In addition, because FH is highly penetrant, once an individual with FH is identified, cascade screening of relatives has been shown to be highly cost-effective in reducing excess morbidity in family members.2,6–8”
    Quote in the paperSubstantial claim1 of 4 sources checked
    • cites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source
    • cites [6] National Collaborating Centre for Primary Care (UK). Clinical guidelines and evidence review for familial hypercholesterolaemia: the identification and management of adults and children with familial hypercholesterolaemia. (Royal College of General Practitioners (UK), 2011).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

    • cites [7] Knowles, J. W., Rader, D. J. & Khoury, M. J. Cascade screening for familial hypercholesterolemia and the use of genetic testing. JAMA 318, 381–382 (2017). source
    • cites [8] Public Health Genomics. Centers for Disease Control and Prevention (2014). Available at: https://www.cdc.gov/genomics/implementation/toolkit/fh_1.htm. (Accessed: 9th December 2017).

      Not checked in the free run.

  • Premisecitation

    Untreated lifelong elevation of low-density lipoprotein cholesterol in individuals with familial hypercholesterolemia causes up to a 20-fold excess risk of atherosclerotic cardiovascular disease versus those with normal LDL-C levels.

    “Lifelong elevation of low-density lipoprotein cholesterol (LDL-C) in individuals with FH cause up to a 20-fold excess risk of atherosclerotic cardiovascular disease (ASCVD) versus those with normal LDL-C levels.2,3”
    Quote in the paperSubstantial claim1 of 2 sources checked
    • cites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source
    • cites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). source

      The Results (Clinical Impact of FH Mutation Identification on CAD Risk; page unavailable) report an OR of 22.3 for coronary artery disease in participants with LDL-C ≥190 mg/dl and an FH mutation, compared with those with LDL-C <130 mg/dl and no mutation. The Discussion describes lifelong LDL-C exposure as a hypothesis explaining the excess risk, not an established cause. The reported outcome is coronary artery disease rather than all ASCVD, and the analysis was not limited to untreated participants. (LLM, conf=0.88)

      Experimental — accuracy not yet measured
  • Premisecitation

    A prevalence of 1:70 is a reasonable estimate for evaluating the classifier in individuals with hypercholesterolemia or increased risk of ASCVD.

    “In our case, we believe that this classifier would be most useful in flagging individuals within the healthcare system with an enhanced risk of ASCVD, we set a prevalence of 1:70 for evaluation, which is a reasonable estimate given the known prevalence of FH in individuals with hypercholesterolemia or increased risk of ASCVD (Supplementary Table 1).3”
    Quote in the paperSubstantial claim1 of 1 sources checked
    • cites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). source

      The Results section (Diagnostic Yield of Gene Sequencing in Severe Hypercholesterolemia; page not provided) reports an FH-mutation prevalence of 1.7% among participants with LDL cholesterol ≥190 mg/dl, reasonably close to 1 in 70. It does not establish that prevalence for the broader group of people with increased ASCVD risk, so the proposed classifier-evaluation prevalence is only partly supported. (LLM, conf=0.82)

      Experimental — accuracy not yet measured
  • Classified as probably not a substantial claim. 19
  • May combine two claims. 15

  • Classified as probably not a substantial claim. May combine two claims.

    Premisecitation

    The DLCN criteria assigns points based on clinical findings, family history, and genetic testing results, and categorizes individuals as definite, probable, possible, or unlikely FH.

    “The DLCN criteria assigns points for based on clinical findings (e.g., LDL-cholesterol levels, history of coronary artery disease, physical exam), family history and genetic testing results (if known). Based on the score, individuals are categorized as definite, probable, possible or unlikely FH.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source

      The Diagnosis section (page unavailable) says the Dutch Lipid Clinic Network criteria use a score combining LDL-C, clinical signs, family history of coronary artery disease, and DNA markers. It says a score above 5 makes FH highly probable, but does not give the claimed four categories—definite, probable, possible, and unlikely. (LLM, conf=0.86)

      Experimental — accuracy not yet measured
    • cites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). source

  • Classified as probably not a substantial claim.

    Study design and conductcitation

    When evaluating identified patients via chart review, individuals were judged to have FH if they met either DLCN or MEDPED criteria.

    “When evaluating the identified patients via chart review, individuals were judged to have FH if they met either DLCN or MEDPED criteria.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source

      The Diagnosis section describes DLCN and MEDPED as established clinical criteria for diagnosing FH. It does not describe evaluating patients by chart review or specify a rule that meeting either set of criteria establishes FH in that review. (LLM, conf=0.84)

      Experimental — accuracy not yet measured
    • cites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). source

      The Diagnosis and Cascade, opportunistic, and universal screening sections recommend DLCN criteria for a clinical FH diagnosis. Our copy does not describe using MEDPED as an alternative or evaluating identified patients by chart review, so it does not support those parts of the sentence. (LLM, conf=0.84)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim. May combine two claims.

    Study design and conductreasoning

    To ensure relevance and avoid trivial classification, non-cases (controls) were limited to patients with more than one visit at Stanford, at least one visit within the last 2 years, and one comorbidity from hypertension, coronary atherosclerosis, dyslipidemia, or myocardial infarction.

    “To ensure relevance of our classifier for use on patients at a higher risk for ASCVD for discerning between FH cases and similar non-FH cases “in the wild”,37,38 and to ensure that trivial classification (e.g., healthy 25-year olds classified as non-FH) does not produce optimistic results, we limited our non-cases (“controls”) to patients that: (1) had more than one visit at Stanford; (2) had at l”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [37] Norén, G. N., Caster, O., Juhlin, K. & Lindquist, M. Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance. Drug Saf. 37, 655–659 (2014). source
    • cites [38] Harpaz, R., DuMouchel, W. & Shah, N. H. Comment on: ‘Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance′. Drug Saf. 38, 113–114 (2015). source

      The citation is attached to the rationale for evaluating a classifier on higher-risk patients ‘in the wild’; the control-selection rules are the citing authors’ own methods. The cited paper’s To the Editor section discusses the real-world relevance of pharmacovigilance benchmarks, but not FH classification, ASCVD risk, Stanford patients, or the stated control criteria. Searches for those terms found no supporting passage in our copy. (LLM, conf=0.78)

      Experimental — accuracy not yet measured

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Premisecitation

    Familial hypercholesterolemia is an autosomal dominant condition with an estimated prevalence of approximately 1 in 250.

    “Familial hypercholesterolemia (FH) is an autosomal dominant condition with an estimated prevalence of approximately 1 in 250,1”
    Quote in the paperNot a substantial claim1 of 1 sources checked
    • cites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source

      The opening section describes FH as an autosomal-dominant genetic disease. It estimates the prevalence of heterozygous FH at approximately 1 in 200 to 1 in 500, which includes 1 in 250 but does not give that figure as a point estimate. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • Classified as probably not a substantial claim.

    Study design and conductown data

    Patients were scored using DLCN or MEDPED criteria to determine the relationship between the machine-learning algorithm and these clinical criteria.

    “Patients were then scored using the DLCN or MEDPED criteria1,5 to determine the relationship between the machine-learning algorithm and these widely used clinical criteria, which are currently used to help inform choices about referral and clinical care.”
    Quote in the paperSubstantial claim2 of 2 sources checked
    • cites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source
    • cites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). source

      The Diagnosis section (page not available) recommends DLCN criteria for clinical diagnosis of familial hypercholesterolaemia, and the Underdiagnosis section provides their scoring scheme. Our copy does not establish the MEDPED criteria or a comparison with a machine-learning algorithm. The patient scoring and comparison appear to describe the citing authors’ own work, rather than a finding attributed to this source. (LLM, conf=0.85)

      Experimental — accuracy not yet measured

  • May combine two claims.

    Findingown data

    The supervised machine learning classifier for finding patients that might have FH using EHR data achieved a positive predictive value of 0.88, sensitivity of 0.75, and specificity of 0.99.

    “In conclusion, we have demonstrated that a supervised machine learning approach to building a classifier for finding patients that might have FH using EHR data is feasible with a positive predictive value of 0.88, sensitivity of 0.75 and specificity of 0.99.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Findingown data

    The classifier obtained a positive predictive value of 0.88 and sensitivity of 0.75 on a held-out test-set.

    “Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 on a held-out test-set.”
    • Also stated in: § Study design · plain
      “Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 (Fig. 1a and Table 1) on a held-out test-set.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductreasoning

    Patients with nephrotic syndrome or obstructive (cholestatic) liver disease were excluded because both cause extremely elevated LDL-cholesterol for different reasons and do not convey the same risk of coronary artery disease.

    “We also excluded patients with nephrotic syndrome, or obstructive (cholestatic) liver disease because both result in extremely elevated LDL-cholesterol levels for entirely different reasons and do not convey the same risk of coronary artery disease.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The classifier was validated by classifying 35,562 patients and reviewing predictions across a range of probability scores via chart review, and by applying DLCN and MEDPED criteria to determine the likelihood of flagged patients having FH.

    “We validated our classifier by classifying 35,562 patients and reviewing predictions across a range of probability scores via chart review, and by applying established criteria, such as DLCN and MEDPED criteria, to determine the likelihood of the patients flagged by our classifier to have FH.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductown data

    The best performing classifier was tested on unseen 17 cases and 1190 controls with 1:70 prevalence.

    “The best performing classifier from this process was tested on the unseen 17 cases and 1190 controls with 1:70 prevalence.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Interpretationreasoning

    Compared to universal genetic testing or clinical criteria-based screening, the economics of EHR-based detection of familial hypercholesterolemia through machine learning are extremely favorable and can massively improve a health system's ability to find patients at risk.

    “Therefore, compared to the implementation of universal genetic testing or clinical criteria-based screening, the economics of EHR-based detection of FH through machine-learning are extremely favorable and can massively improve the ability of a health system to find patients at risk.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductnone given

    Directly classifying the Geisinger dataset with its native 466:5000 case-to-non-case ratio would lead to overly optimistic estimates of performance.

    “Directly classifying the Geisinger dataset with its native 466:5000 case to non-case ratio would lead to overly optimistic estimates of performance.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductown data

    The classifier was built using both structured and unstructured EHR data from Stanford and validated with data from an independent site (Geisinger) including a subset of genetically confirmed cases.

    “Our classifier was built using both structured (e.g., labs, procedures, diagnostic codes) and unstructured (e.g., text from clinical notes and radiology reports) EHR data from Stanford and validated with data from an independent site (Geisinger) including a subset of genetically confirmed cases (Fig. 1).”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductcitation

    The Geisinger validation dataset contains 466 FH cases diagnosed using DLCN criteria, of which 236 were confirmed by genetic testing.

    “The data contain 466 FH cases that have been diagnosed using the DLCN criteria,2 of which 236 FH cases were confirmed by genetic testing.”
    Quote in the paperSubstantial claim0 of 1 sources checked
    • cites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source

  • May combine two claims.

    Study design and conductown data

    The random forest classifier was applied to held out patients from Stanford to flag potential FH cases, and predictions were evaluated with manual chart review.

    “As shown in Fig. 1b, we applied the random forest classifier to held out patients from Stanford to flag potential FH cases, and evaluated the predictions with manual chart review.”
    Quote in the paperSubstantial claim

  • May combine two claims.

    Study design and conductnone given

    The Stanford test dataset used an estimated familial hypercholesterolemia prevalence of 1:70, and if the real-life prevalence in a lipid clinic is drastically different, classifier performance will differ.

    “For the Stanford test dataset, we used an (estimated) FH prevalence of 1:70 in the test-set, if the real-life prevalence of FH cases in a lipid clinic is drastically different the classifier performance will differ.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductreasoning

    Measuring downstream outcomes such as overall rates of familial hypercholesterolemia diagnosis, cascade screening, atherosclerotic cardiovascular disease events, or survival was not addressed in this study.

    “Measuring downstream outcomes such as the overall rates of FH diagnosis, cascade screening, ASCVD events or survival would require longitudinal observation after implementation of this algorithm and was not addressed in this study.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductnone given

    The entire process was repeated 10 times with different splits of the train/test sets.

    “We repeated the entire process 10 times (choosing different splits of the train/test sets).”
    Quote in the paperNot a substantial claim

  • May combine two claims.

    Interpretationreasoning

    Applied broadly, using the classifier to screen using EHRs could identify many thousands of undiagnosed FH patients and lead to more effective therapy and screening of their families.

    “Applied broadly, using our classifier to screen using EHRs could identify many thousands of the undiagnosed patients with FH and lead to more effective therapy and screening of their families.”
    Quote in the paperSubstantial claim
    • Rests on an assumption: Proven efficacious interventions already exist that can prevent catastrophic cardiovascular events in patients with FH.
    • Rests on an assumption: A found FH case can be used to cascade to find multiple other cases within an extended family.

  • Classified as probably not a substantial claim.

    Study design and conductown data

    For external validation, the Stanford classifier was applied to unseen data from Geisinger.

    “For external validation, we applied our Stanford classifier to unseen data from Geisinger.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Study design and conductown data

    The classifier was trained using 10-fold cross-validation on the 1:30 prevalence training set for parameter tuning.

    “We trained the classifier using 10-fold cross-validation, on the 1:30 prevalence training set, for parameter tuning.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    The MEDPED criteria are based on age-adjusted cholesterol levels and family pedigree information, with cutoffs designed to identify individuals who would have a causal mutation found on genetic testing.

    “The MEDPED criteria are based on age-adjusted cholesterol levels and factor into account family pedigree information (if known) with cutoffs designed to identify individuals who would have a causal mutation found on genetic testing.33”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [33] Haase, A. & Goldberg, A. C. Identification of people with heterozygous familial hypercholesterolemia. Curr. Opin. Lipidol. 23, 282–289 (2012). source

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    In previous work, a logistic regression classifier was built and compared with the random forest classifier.

    “In previous work,40 we built and compared a logistic regression classifier with the random forest classifier.”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [40] Niehaus, K. E., Banda, J. M., Knowles, J. W. & Shah, N. H. FIND FH—A phenotype model to identify patients with familial hypercholesterolemia. in Proceedings of Data Mining for Medical Informatics Workshop 2015 (2015).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Interpretationnone given

    Having more training data would probably build a better classifier.

    “and as with most machine learning approaches, having more training data would probably build a better classifier.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing).

    Premisecitation

    Upsampling or downsampling is a commonly used technique in machine learning with severe class imbalance.

    “Such upsampling (or downsampling) is a commonly used technique in machine learning with severe class imbalance.39”
    Quote in the paperNot a substantial claim0 of 1 sources checked
    • cites [39] Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J. Data Mining: Practical Machine Learning Tools and Techniques. (Morgan Kaufmann, 2016).

      Couldn’t be matched to a paper in Crossref, so it can’t be checked.

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    Before training, cases and controls were matched by age, gender, and length of record to avoid trivial classification.

    “Before training the classifier, we matched cases and controls by age, gender, and length of record, in order to avoid trivial classification.”
    Quote in the paperSubstantial claim

  • Classified as probably not a substantial claim.

    Study design and conductreasoning

    A subset of 71 Geisinger FH cases and 4970 controls in a 1:70 ratio was used to assess the ability of the Stanford classifier to discriminate true cases from non-cases.

    “Therefore, we used a subset (n = 71) of the 466 Geisinger FH cases and controls (n = 4970) in a 1:70 ratio, to assess the ability of the classifier trained at Stanford to discriminate true cases from non-cases.”
    Quote in the paperSubstantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductnone given

    The study complied with all relevant ethical regulations and was approved by Stanford University's institutional review board with waiver of informed patient consent.

    “For all patient data, we have complied with all relevant ethical regulations and the study was approved by Stanford University′s institutional review board with waiver of informed patient consent.”
    Quote in the paperNot a substantial claim

  • Probably not a substantial claim: may not pass S3 (load-bearing). May combine two claims.

    Study design and conductown data

    The model building was performed with de-identified data and only the chart review team had access to medical records.

    “The model building was performed with de-identified data and only the members of the chart review team were provided with access to medical records.”
    Quote in the paperNot a substantial claim
29 of 29

  • The paper relies on this without support.

    Current guideline-recommended diagnostic strategies for familial hypercholesterolemia largely rely on the availability of untreated LDL-C values and adequate family history information, either or both of which are often unavailable to healthcare providers.

    “In addition, this strategy largely relies on the availability of untreated LDL-C values and adequate family history information, either/both of which are often unavailable to the healthcare provider.”

  • The paper relies on this without support.

    The Geisinger familial hypercholesterolemia cases and matched non-cases are correctly labeled as familial hypercholesterolemia and non-familial hypercholesterolemia.

    “We validated our classifier and its ability to detect FH patients by evaluating it on an independent dataset from Geisinger”
    “466 cases provided by Geisinger”

  • The paper relies on this without support.

    Chart review using DLCN and MEDPED criteria correctly identifies likely familial hypercholesterolemia cases.

    “47/56 have a DLCN score of >=3 or are MEDPED positive (84%)”

  • The paper relies on this without support.

    Proven efficacious interventions already exist that can prevent catastrophic cardiovascular events in patients with FH.

    “Such case finding is particularly relevant because once a case is found, proven efficacious interventions already exist that can prevent catastrophic cardiovascular events; furthermore, that case can be used to ‘cascade′ to find multiple other cases within an extended family.”

  • The paper relies on this without support.

    A found FH case can be used to cascade to find multiple other cases within an extended family.

    “Such case finding is particularly relevant because once a case is found, proven efficacious interventions already exist that can prevent catastrophic cardiovascular events; furthermore, that case can be used to ‘cascade′ to find multiple other cases within an extended family.”

  • The paper relies on this without support.

    The Geisinger Healthcare System data are sufficiently representative of other health systems for the classifier's external validity to generalize.

    “We used unseen FH and non-FH patient data from the Geisinger Healthcare System to demonstrate external validity of the classifier.”
6 of 6

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

  • Genetic-testing cost conflicts with the stated eight-in-ten yield

    Major

    At $1,000 per test and an eight-in-ten chance of FH among flagged patients, the expected testing cost per case found is $1,250, not $1,429.

    Reported 1429 · recomputed 1250

  • Internal PPV is incompatible with reported sensitivity and specificity

    Major

    With 17 cases and 1,190 controls, sensitivity of 0.75 and specificity of 0.99 imply a PPV of about 0.52, not 0.88. Even allowing for rounding of the reported specificity, the PPV cannot reach 0.88.

  • External PPV is incompatible with reported sensitivity and specificity

    Major

    With 71 cases and 4,970 controls, sensitivity of 0.68 and specificity of 0.99 imply a PPV of about 0.49, not 0.85. Rounding specificity to two decimal places cannot account for that difference.

  • Clinician time conflicts with the stated eight-in-ten yield

    Minor

    At 15 minutes per evaluation and an eight-in-ten chance of FH among flagged patients, the expected time per case found is 18.75 minutes, not 21.4 minutes.

    Reported 21.4 · recomputed 18.75

4 of 4

Cites

Flagged first, then by citations

Research works printed but not linked: 4

  1. National Collaborating Centre for Primary Care (UK). Clinical guidelines and evidence review for familial hypercholesterolaemia: the identification and management of adults and children with familial hypercholesterolaemia. (Royal College of General Practitioners (UK), 2011).
  2. Wagstaff, K. Machine Learning that Matters. arXiv [cs.LG] (2012).
  3. Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J. Data Mining: Practical Machine Learning Tools and Techniques. (Morgan Kaufmann, 2016).
  4. Niehaus, K. E., Banda, J. M., Knowles, J. W. & Shah, N. H. FIND FH—A phenotype model to identify patients with familial hypercholesterolemia. in Proceedings of Data Mining for Medical Informatics Workshop 2015 (2015).

Cited by

Sorted by citations

Related

Authors
Banda, J. M., Sarraju, A., Abbasi, F., Parizo, J., Pariani, M., Ison, H., Briskin, E., Wand, H., Dubois, S., Jung, K., Myers, S. A., Rader, D. J., Leader, J. B., Murray, M. F., Myers, K. D., Wilemon, K., Shah, N. H., & Knowles, J. W.
Affiliations
  • Georgia State University (US)
  • Stanford Cardiovascular Institute (US)
  • Stanford University (US)
  • FH Foundation (US)
  • Geisinger Health System (US)
  • Yale University (US)
Corresponding author
  • Shah, Nigam H.
  • Knowles, Joshua W.
ORCID
Venue
npj Digital Medicine
Publisher
Springer Science and Business Media LLC
Published
2019-04-11
Volume
2
Issue
1
Pages
23-23
ISSN
2398-6352 (ISSN-L)
DOI
10.1038/s41746-019-0101-5
PMC
PMC6550268
OpenAlex
W2935996710
Cited by
123
References
38
Citations by year
2026: 7 · 2025: 20 · 2024: 14 · 2023: 18 · 2022: 21 · 2021: 22 · 2020: 14 · 2019: 5 · 2018: 2
FWCI
7.06
Type
journal article
Language
en
Open access
Licence
Locations
  • npj Digital Medicine (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • Europe PMC (PubMed Central) (submittedVersion, cc-by)
  • PubMed Central (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
Funders
American Heart Association (15IRG22930034)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
  • Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
  • Health Sciences › Medicine › Surgery › Lipoproteins and Cardiovascular Health
Keywords
familial hypercholesterolemia, missed cases, electronic health records, machine learning classifiers, random forest, underdiagnosis, coronary artery disease risk, external validation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Claims read by deepseek-v4.1-flash under definition version 3 (extract-v3) on 2026-10-27.

Classified by jev-1.13-20260917 (gold-guideline-v0.2) on 2026-10-27.

Cited sources are checked by gpt-6-sol.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review