The DOI matches its Crossref record; 38 of the 38 references it cites were found in a registry or an index; 22 of the 36 sources its claims cite were checked for support.
Finding missed cases of familial hypercholesterolemia in health systems using machine learning
Open accessAbstract
Familial hypercholesterolemia (FH) is an underdiagnosed dominant genetic condition affecting approximately 0.4% of the population and has up to a 20-fold increased risk of coronary artery disease if untreated. Simple screening strategies have false positive rates greater than 95%. As part of the FH Foundation′s FIND FH initiative, we developed a classifier to identify potential FH patients using electronic health record (EHR) data at Stanford Health Care. We trained a random forest classifier using data from known patients ( n = 197) and matched non-cases ( n = 6590). Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 on a held-out test-set. We evaluated the accuracy of the classifier′s predictions by chart review of 100 patients at risk of FH not included in the original dataset. The classifier correctly flagged 84% of patients at the highest probability threshold, with decreasing performance as the threshold lowers. In external validation on 466 FH patients (236 with genetically proven FH) and 5000 matched non-cases from the Geisinger Healthcare System our FH classifier achieved a PPV of 0.85. Our EHR-derived FH classifier is effective in finding candidate patients for further FH screening. Such machine learning guided strategies can lead to effective identification of the highest risk patients for enhanced management strategies.
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
A random forest classifier using structured and unstructured EHR data from Stanford identified patients likely to have familial hypercholesterolemia, with internal PPV 0.88 and sensitivity 0.75 and external Geisinger PPV 0.85. Chart review of high-probability flags found 84% met DLCN ≥3 or MEDPED criteria, suggesting the model can prioritize undiagnosed FH cases for further evaluation.
Key findings
- On a held-out Stanford test set, the classifier achieved PPV 0.88, sensitivity 0.75, specificity 0.99, F1 0.81, AUROC 0.94, and AUPRC 0.71.
- External validation at Geisinger achieved PPV 0.85, sensitivity 0.68, specificity 0.99, F1 0.75, AUROC 0.94, and AUPRC 0.68; limiting to genetically confirmed cases gave an F1 score of 0.82.
- In chart review of 100 held-out patients, among 56 with classifier probability 0.99–0.90, 47/56 (84%) had DLCN ≥3 or were MEDPED positive; only 3 of the flagged chart-review-confirmed likely FH patients had been clinically diagnosed with FH.
- Top classifier features included LDL cholesterol text mentions, high/very high cholesterol and LDL lab bins, cardiology clinic mentions, atorvastatin/ezetimibe/rosuvastatin prescriptions, and hyperlipidemia diagnosis codes.
Methods
- A random forest classifier was built using OMOP-CDM/APHRODITE with structured (labs, diagnoses, drugs) and unstructured (clinical notes) EHR data; features were normalized by follow-up length, rare features removed, and lab values binned into five categories.
- Stanford training used 197 known FH patients and matched non-cases (n = 6,590 total: 5,400 training controls and 1,190 test controls); cases were split into 180 training and 17 test, with 1:70 test prevalence, 10-fold cross-validation, and repeated train/test splits; chart review of 100 held-out flagged patients used DLCN/MEDPED criteria.
- External validation used Geisinger data with 466 FH cases (236 genetically confirmed) and 5,000 matched non-cases; 71 cases and 4,970 controls were sampled at 1:70 prevalence and evaluated over 10 iterations.
Limitations
- Stated: the Stanford test set assumed 1:70 FH prevalence, so performance may differ if real lipid-clinic prevalence differs; downstream outcomes such as diagnosis rates, cascade screening, ASCVD events, or survival were not assessed; the classifier was not trained on pooled data because that would prevent external-validity assessment; and the positive sample was relatively small (n = 197 at Stanford).
- Stated: cost-effectiveness discussion assumes follow-up costs for chart review and genetic testing, and the authors prioritized PPV over recall because those follow-up actions are expensive.
- Evident from the text: training was retrospective and single-site, while chart review and external validation used sampled case-control ratios rather than full native ratios, so estimates depend on the chosen prevalence; top features include non-traditional variables (e.g., paroxysmal supraventricular tachycardia, “Cytologic”) whose clinical relevance or confounding is not resolved.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
19 Own method · 19 Own result · 12 Partly supported · 10 Own interpretation · 3 Sources not found · 2 No full text of the sources · 2 No basis given · 1 Supported · 1 Supported so far · accuracy not yet measured
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“it is estimated that less than 10% of persons with FH in the US have been diagnosed,1”
- Also stated in: § Discussion · hedged
“It is estimated that only ~10% of patients with FH are diagnosed in the United States.7”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Partly supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). sourceExperimental — accuracy not yet measured
The introductory section says FH remains underdiagnosed worldwide, which supports the general point. The Diagnosis and Screening sections discuss barriers to recognition, but the passages do not give a US estimate or say that fewer than 10% have been diagnosed. Page numbers were not supplied. (LLM, conf=0.83)
- Supportedcites [7] Knowles, J. W., Rader, D. J. & Khoury, M. J. Cascade screening for familial hypercholesterolemia and the use of genetic testing. JAMA 318, 381–382 (2017). source
The Clinical and Public Health Considerations section states that current estimates put the proportion of FH cases identified in the United States at less than 10%. This supports the citing sentence’s estimated proportion diagnosed. (LLM, conf=0.94)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Discussion · hedged
Every source the paper cites for this claim was checked and supports it.
In the paper
“While high LDL-C is a cardinal feature of FH, less than 5% of adults with an LDL-C > 190 mg/dl will be found to harbor a causal FH gene mutation.3”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Supportedcites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). source
The Results section reports that 16 of 956 participants (1.7%) with LDL-C >190 mg/dl in five prospective cohorts harbored an FH mutation. The combined analysis likewise found mutations in 1.7% of participants with LDL-C ≥190 mg/dl, supporting the claim that the proportion is below 5%. (LLM, conf=0.94)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The sources with a verdict support this claim; the rest weren’t checked yet, have no full text we could read, or their verdicts aren’t rated yet.
In the paper
“In addition, because FH is highly penetrant, once an individual with FH is identified, cascade screening of relatives has been shown to be highly cost-effective in reducing excess morbidity in family members.2,6–8”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source
- Not matchedcites [6] National Collaborating Centre for Primary Care (UK). Clinical guidelines and evidence review for familial hypercholesterolaemia: the identification and management of adults and children with familial hypercholesterolaemia. (Royal College of General Practitioners (UK), 2011).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
- Supportedcites [7] Knowles, J. W., Rader, D. J. & Khoury, M. J. Cascade screening for familial hypercholesterolemia and the use of genetic testing. JAMA 318, 381–382 (2017). source
The Cost-Effectiveness of FH Cascade Screening section reports that cascade testing is cost-effective, with an estimated $2,500–$4,500 per quality-adjusted life-year gained. The How Cascade Screening for FH Works section describes identifying affected relatives and increasing treatment, while the Bottom Line says cascade screening can reduce ASCVD morbidity and mortality. Our copy does not provide page numbers for these passages. (LLM, conf=0.89)
- Not checked yetcites [8] Public Health Genomics. Centers for Disease Control and Prevention (2014). Available at: https://www.cdc.gov/genomics/implementation/toolkit/fh_1.htm. (Accessed: 9th December 2017).
boosteddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Lifelong elevation of low-density lipoprotein cholesterol (LDL-C) in individuals with FH cause up to a 20-fold excess risk of atherosclerotic cardiovascular disease (ASCVD) versus those with normal LDL-C levels.2,3”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Not checked — no full text of the sourcecites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source
- Partly supportedcites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). sourceExperimental — accuracy not yet measured
The Results (Clinical Impact of FH Mutation Identification on CAD Risk; page unavailable) report an OR of 22.3 for coronary artery disease in participants with LDL-C ≥190 mg/dl and an FH mutation, compared with those with LDL-C <130 mg/dl and no mutation. The Discussion describes lifelong LDL-C exposure as a hypothesis explaining the excess risk, not an established cause. The reported outcome is coronary artery disease rather than all ASCVD, and the analysis was not limited to untreated participants. (LLM, conf=0.88)
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“In our case, we believe that this classifier would be most useful in flagging individuals within the healthcare system with an enhanced risk of ASCVD, we set a prevalence of 1:70 for evaluation, which is a reasonable estimate given the known prevalence of FH in individuals with hypercholesterolemia or increased risk of ASCVD (Supplementary Table 1).3”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Partly supportedcites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). sourceExperimental — accuracy not yet measured
The Results section (Diagnostic Yield of Gene Sequencing in Severe Hypercholesterolemia; page not provided) reports an FH-mutation prevalence of 1.7% among participants with LDL cholesterol ≥190 mg/dl, reasonably close to 1 in 70. It does not establish that prevalence for the broader group of people with increased ASCVD risk, so the proposed classifier-evaluation prevalence is only partly supported. (LLM, conf=0.82)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Simple screening strategies have false positive rates greater than 95%.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Abstract
What the sources say
- Partly supportedcites [3] Khera, A. V. et al. Diagnostic yield and clinical utility of sequencing familial hypercholesterolemia genes in patients with severe hypercholesterolemia. J. Am. Coll. Cardiol. 67, 2578–2589 (2016). sourceExperimental — accuracy not yet measured
The Results section (page not available) reports that only 24 of 1,386 people with LDL cholesterol ≥190 mg/dl carried an FH mutation; thus, more than 98% identified by this simple LDL-based threshold did not carry one. The paper does not establish a formal false-positive rate or evaluate simple screening strategies generally: the reported percentage is the proportion of screen-positive people without an identified mutation. (LLM, conf=0.86)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Those with higher scores are more likely to have a causal mutation identified on genetic testing.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
What the sources say
- Supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source
The Diagnosis section (page not available in our copy) says the Dutch Lipid Clinic Network scoring system makes FH highly probable above a score of 5 and describes the criteria as predicting potential FH mutations. It also reports higher genetic-testing yields in patients with tendon xanthomata, a clinical sign used in the scoring system. Together, these passages support the claimed association, although the paper does not give mutation-detection rates by DLCN score. (LLM, conf=0.84)
- Partly supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). sourceExperimental — accuracy not yet measured
The Diagnosis section recommends genetic testing for people with definite or probable FH (DLCN >5), particularly those with an obvious clinical diagnosis. The Clinical vs. mutation diagnosis section says that 10–40% of people with a clinical diagnosis lack a detectable causal mutation, depending on referral criteria. These passages support prioritizing higher-scoring patients for testing, but do not establish that mutation detection becomes more likely as DLCN scores rise; page numbers are unavailable in our copy. (LLM, conf=0.78)
hedgedcorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The FH patients were followed at an FH-specific clinic and had been diagnosed as probable or definite FH using existing diagnostic criteria including genetic testing information when available.5,33–36”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Partly supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). sourceExperimental — accuracy not yet measured
The Diagnosis and Underdiagnosis sections (page unavailable) describe criteria for probable or definite FH and include molecular genetic testing in the diagnostic framework. The paper does not establish that the particular patients used to build the classifier were diagnosed under those criteria or that genetic information was available for them. (LLM, conf=0.84)
- Not checked — no full text of the sourcecites [33] Haase, A. & Goldberg, A. C. Identification of people with heterozygous familial hypercholesterolemia. Curr. Opin. Lipidol. 23, 282–289 (2012). source
- Not checked — no full text of the sourcecites [34] Civeira, F., International Panel on Management of Familial Hypercholesterolemia. Guidelines for the diagnosis and management of heterozygous familial hypercholesterolemia. Atherosclerosis 173, 55–68 (2004). source
- Not checked — no full text of the sourcecites [35] Austin, M. A., Hutter, C. M., Zimmern, R. L. & Humphries, S. E. Genetic causes of monogenic heterozygous familial hypercholesterolemia: a HuGE prevalence review. Am. J. Epidemiol. 160, 407–420 (2004). source
- Not checked — no full text of the sourcecites [36] Williams, R. R. et al. Diagnosing heterozygous familial hypercholesterolemia using new practical criteria validated by molecular genetics. Am. J. Cardiol. 72, 171–176 (1993). source
plainn = 197- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Importantly, the risk of ASCVD can be largely ameliorated through early identification and treatment with lipid-lowering therapies.1,4–6”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source
The opening section says that identifying people with familial hypercholesterolemia at a young age is a priority because treatment can lower LDL cholesterol and prevent or delay ischemic heart disease. It also describes a dramatic improvement in life expectancy with effective cholesterol lowering. The Pharmacotherapy section recommends starting lipid-lowering therapy at diagnosis and as early as possible. (LLM, conf=0.91)
- Supportedcites [4] Besseling, J. et al. Selection of individuals for genetic testing for familial hypercholesterolaemia: development and external validation of a prediction model for the presence of a mutation causing familial hypercholesterolaemia. Eur. Heart J. 38, 565–573 (2017). source
The Introduction (page not specified in our copy) says statin therapy robustly improves cardiovascular risk, especially when started young, and identifies early detection as crucial. This supports the claim about early identification and lipid-lowering treatment in familial hypercholesterolaemia. (LLM, conf=0.90)
- Supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). source
The Treatment section (page not supplied) says statins reduce coronary heart disease events in people with familial hypercholesterolaemia and that treatment before disease onset can bring coronary-heart-disease-free survival close to that of the general population. The Introduction and Children with familial hypercholesterolaemia sections also link diagnosis and earlier treatment to preventing atherosclerosis and improving outcomes. (LLM, conf=0.92)
- Not matchedcites [6] National Collaborating Centre for Primary Care (UK). Clinical guidelines and evidence review for familial hypercholesterolaemia: the identification and management of adults and children with familial hypercholesterolaemia. (Royal College of General Practitioners (UK), 2011).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This classifier achieved a PPV of 0.83, sensitivity of 0.66 and specificity of 0.99.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Sensitivity | 0.75 | 0.68 (0.002) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For instance, the top 20 predictive concepts in our random forest classifier include variables not used in the traditional FH clinical criteria, such as the diagnosis codes for paroxysmal supraventricular tachycardia and triglyceride levels (Table 2).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For completeness, we also built a classifier using Geisinger data (which had 0.88 PPV, 0.74 sensitivity, and 0.99 specificity)”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For completeness, we also built a classifier using Geisinger data (which had 0.88 PPV, 0.74 sensitivity, and 0.99 specificity)”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The classifier uses a combination of laboratory tests, text mentions, diagnosis codes and prescriptions as the top features with many of them being related to cholesterol levels and hyperlipidemia diagnosis.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“As expected, there is a slight drop in PPV, AUROC, and AUPRC, given the different operational practices between institutions (coding, labs, clinical narratives).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“One thing to note is that all the laboratory tests selected as informative features correspond to the high and very high bins created during feature engineering, which capture the higher than usual total cholesterol and LDL-C measurements.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| Specificity | 0.99 | 0.99 (0.001) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Indeed, among the patients flagged by our random forest classifier for whom chart review confirmed a likely diagnosis of FH, only 3 had been clinically diagnosed with FH.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| AUPRC | 0.71 | 0.68 (0.054) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| PPV | 0.88 | 0.85 (0.002) |”
- Also stated in: § Abstract · plain
“our FH classifier achieved a PPV of 0.85.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
plainn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Also stated in: § Abstract · plain
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For completeness, we also built a classifier using Geisinger data (which had 0.88 PPV, 0.74 sensitivity, and 0.99 specificity)”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plainn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“| AUROC | 0.94 | 0.94 (0.003) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This classifier achieved a PPV of 0.83, sensitivity of 0.66 and specificity of 0.99.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In other words, 47/56 have a DLCN score of >=3 or are MEDPED positive (84%).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Evaluation via chart review
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: Chart review using DLCN and MEDPED criteria correctly identifies likely familial hypercholesterolemia cases.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This classifier achieved a PPV of 0.83, sensitivity of 0.66 and specificity of 0.99.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, the rate of likely cases diminishes in lower probability bins especially in those with a probability score < 0.7”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Evaluation via chart review
plaincorrelationaln = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Limiting the analysis to just the genetically confirmed cases has an F1 score of 0.82.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our EHR-derived FH classifier is effective in finding candidate patients for further FH screening.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Abstract
plaindirectn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Such machine learning guided strategies can lead to effective identification of the highest risk patients for enhanced management strategies.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Abstract
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, after applying EHR-based screening, the chance that an individual flagged by our algorithm has FH is 8 out of 10.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As a result of this massive chance in post-test prevalence, the cost to find one new case drops to $1429 in genetic counseling and testing, and 21.4 min of clinician time.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“The overall performance was lower than the random forest, and in the current work, we only use the random forest classifier.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Not matchedcites [40] Niehaus, K. E., Banda, J. M., Knowles, J. W. & Shah, N. H. FIND FH—A phenotype model to identify patients with familial hypercholesterolemia. in Proceedings of Data Mining for Medical Informatics Workshop 2015 (2015).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Finally, our sample of positive FH patients is relatively small (n = 197 at Stanford)”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plainn = 197- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We used unseen FH and non-FH patient data from the Geisinger Healthcare System to demonstrate external validity of the classifier.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
boostedn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Note that directly classifying the external dataset with its native 466:5000 case to non-case ratio would lead to overly optimistic estimates of PPV.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
hedgeddirectn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“As expected, there is a slight drop in PPV, AUROC, and AUPRC, given the different operational practices between institutions (coding, labs, clinical narratives).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plaindirectn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to universal genetic testing or clinical criteria-based screening of all comers, the use of EHR-based detection of FH through machine-learning can massively improve the ability of a health system to find patients at risk of FH.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaindirectn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The Geisinger Healthcare System data are sufficiently representative of other health systems for the classifier's external validity to generalize.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The results show that the classifier trained with Stanford data has good external validity in identifying FH patients at a different institution.”
- Also stated in: § Discussion · hedged
“These findings point to the external validity and overall potential utility of our EHR-based classifier to screen for patients with a high probability of FH.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
plaincorrelationaln = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Rests on an assumption: The Geisinger familial hypercholesterolemia cases and matched non-cases are correctly labeled as familial hypercholesterolemia and non-familial hypercholesterolemia.
- Also stated in: § Discussion · hedged
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We did not train using pooled data because doing so would not allow the assessment of external validity of the classifier.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Probably not a substantial claim. 19
- It may state two separate claims as one. 15
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“The DLCN criteria assigns points for based on clinical findings (e.g., LDL-cholesterol levels, history of coronary artery disease, physical exam), family history and genetic testing results (if known). Based on the score, individuals are categorized as definite, probable, possible or unlikely FH.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
What the sources say
- Partly supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). sourceExperimental — accuracy not yet measured
The Diagnosis section (page unavailable) says the Dutch Lipid Clinic Network criteria use a score combining LDL-C, clinical signs, family history of coronary artery disease, and DNA markers. It says a score above 5 makes FH highly probable, but does not give the claimed four categories—definite, probable, possible, and unlikely. (LLM, conf=0.86)
- Supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). source
Table 1 (Underdiagnosis; page not provided) assigns DLCN points for family history, clinical history, physical examination, LDL cholesterol, and a causative mutation found by genetic testing. Its accompanying note gives the score ranges for definite, probable, possible, and unlikely FH. (LLM, conf=0.95)
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“When evaluating the identified patients via chart review, individuals were judged to have FH if they met either DLCN or MEDPED criteria.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
What the sources say
- Partly supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). sourceExperimental — accuracy not yet measured
The Diagnosis section describes DLCN and MEDPED as established clinical criteria for diagnosing FH. It does not describe evaluating patients by chart review or specify a rule that meeting either set of criteria establishes FH in that review. (LLM, conf=0.84)
- Partly supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). sourceExperimental — accuracy not yet measured
The Diagnosis and Cascade, opportunistic, and universal screening sections recommend DLCN criteria for a clinical FH diagnosis. Our copy does not describe using MEDPED as an alternative or evaluating identified patients by chart review, so it does not support those parts of the sentence. (LLM, conf=0.84)
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“To ensure relevance of our classifier for use on patients at a higher risk for ASCVD for discerning between FH cases and similar non-FH cases “in the wild”,37,38 and to ensure that trivial classification (e.g., healthy 25-year olds classified as non-FH) does not produce optimistic results, we limited our non-cases (“controls”) to patients that: (1) had more than one visit at Stanford; (2) had at l”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Supportedcites [37] Norén, G. N., Caster, O., Juhlin, K. & Lindquist, M. Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance. Drug Saf. 37, 655–659 (2014). source
The citation is attached to the aim of evaluating a classifier ‘in the wild,’ not to the citing authors’ subsequent Stanford control-selection rules. The Introduction uses the zoo-versus-savannah analogy to warn that performance on an easy benchmark cannot be extrapolated to a real-world setting, and the Conclusions call for a training ground that reflects prospective real-world surveillance. The Stanford visit and comorbidity criteria are the citing authors’ own methods. (LLM, conf=0.87)
- Could not verifycites [38] Harpaz, R., DuMouchel, W. & Shah, N. H. Comment on: ‘Zoo or savannah? Choice of training ground for evidence-based pharmacovigilance′. Drug Saf. 38, 113–114 (2015). sourceExperimental — accuracy not yet measured
The citation is attached to the rationale for evaluating a classifier on higher-risk patients ‘in the wild’; the control-selection rules are the citing authors’ own methods. The cited paper’s To the Editor section discusses the real-world relevance of pharmacovigilance benchmarks, but not FH classification, ASCVD risk, Stanford patients, or the stated control criteria. Searches for those terms found no supporting passage in our copy. (LLM, conf=0.78)
plainn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may state two separate claims as one.
- Supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Familial hypercholesterolemia (FH) is an autosomal dominant condition with an estimated prevalence of approximately 1 in 250,1”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Introduction
What the sources say
- Partly supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). sourceExperimental — accuracy not yet measured
The opening section describes FH as an autosomal-dominant genetic disease. It estimates the prevalence of heterozygous FH at approximately 1 in 200 to 1 in 500, which includes 1 in 250 but does not give that figure as a point estimate. (LLM, conf=0.85)
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
- Partly supported
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Patients were then scored using the DLCN or MEDPED criteria1,5 to determine the relationship between the machine-learning algorithm and these widely used clinical criteria, which are currently used to help inform choices about referral and clinical care.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
What the sources say
- Supportedcites [1] Gidding, S. S. et al. The agenda for familial hypercholesterolemia: a scientific statement from the american heart association. Circulation 132, 2167–2192 (2015). source
The Diagnosis section describes the Dutch Lipid Clinic Network and Make Early Diagnosis to Prevent Early Deaths (MEDPED) systems as established clinical criteria for diagnosing familial hypercholesterolemia. The patient scoring and machine-learning comparison are the citing authors’ methods, not findings attributed to this source. (LLM, conf=0.90)
- Partly supportedcites [5] Nordestgaard, B. G. et al. Familial hypercholesterolaemia is underdiagnosed and undertreated in the general population: guidance for clinicians to prevent coronary heart disease: consensus statement of the European Atherosclerosis Society. Eur. Heart J. 34, 3478–90a (2013). sourceExperimental — accuracy not yet measured
The Diagnosis section (page not available) recommends DLCN criteria for clinical diagnosis of familial hypercholesterolaemia, and the Underdiagnosis section provides their scoring scheme. Our copy does not establish the MEDPED criteria or a comparison with a machine-learning algorithm. The patient scoring and comparison appear to describe the citing authors’ own work, rather than a finding attributed to this source. (LLM, conf=0.85)
plaincorrelationaln = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
- Supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“In conclusion, we have demonstrated that a supervised machine learning approach to building a classifier for finding patients that might have FH using EHR data is feasible with a positive predictive value of 0.88, sensitivity of 0.75 and specificity of 0.99.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
boostedn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 on a held-out test-set.”
- Also stated in: § Study design · plain
“Our classifier obtained a positive predictive value (PPV) of 0.88 and sensitivity of 0.75 (Fig. 1a and Table 1) on a held-out test-set.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Abstract
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Also stated in: § Study design · plain
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We also excluded patients with nephrotic syndrome, or obstructive (cholestatic) liver disease because both result in extremely elevated LDL-cholesterol levels for entirely different reasons and do not convey the same risk of coronary artery disease.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We validated our classifier by classifying 35,562 patients and reviewing predictions across a range of probability scores via chart review, and by applying established criteria, such as DLCN and MEDPED criteria, to determine the likelihood of the patients flagged by our classifier to have FH.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plainn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The best performing classifier from this process was tested on the unseen 17 cases and 1190 controls with 1:70 prevalence.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
plainn = 17- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, compared to the implementation of universal genetic testing or clinical criteria-based screening, the economics of EHR-based detection of FH through machine-learning are extremely favorable and can massively improve the ability of a health system to find patients at risk.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Directly classifying the Geisinger dataset with its native 466:5000 case to non-case ratio would lead to overly optimistic estimates of performance.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > External validation
hedgeddirectn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Our classifier was built using both structured (e.g., labs, procedures, diagnostic codes) and unstructured (e.g., text from clinical notes and radiology reports) EHR data from Stanford and validated with data from an independent site (Geisinger) including a subset of genetically confirmed cases (Fig. 1).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Study design
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
In the paper
“The data contain 466 FH cases that have been diagnosed using the DLCN criteria,2 of which 236 FH cases were confirmed by genetic testing.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
What the sources say
- Not checked — no full text of the sourcecites [2] Abul-Husn, N. S. et al. Genetic identification of familial hypercholesterolemia within a single U.S. health care system. Science 354, aaf7000 (2016). source
plainn = 466- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“As shown in Fig. 1b, we applied the random forest classifier to held out patients from Stanford to flag potential FH cases, and evaluated the predictions with manual chart review.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
plainn = 100- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For the Stanford test dataset, we used an (estimated) FH prevalence of 1:70 in the test-set, if the real-life prevalence of FH cases in a lipid clinic is drastically different the classifier performance will differ.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Measuring downstream outcomes such as the overall rates of FH diagnosis, cascade screening, ASCVD events or survival would require longitudinal observation after implementation of this algorithm and was not addressed in this study.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We repeated the entire process 10 times (choosing different splits of the train/test sets).”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
plainn = 17- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Applied broadly, using our classifier to screen using EHRs could identify many thousands of the undiagnosed patients with FH and lead to more effective therapy and screening of their families.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
hedgeddirectn = 35,562- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Proven efficacious interventions already exist that can prevent catastrophic cardiovascular events in patients with FH.
- Rests on an assumption: A found FH case can be used to cascade to find multiple other cases within an extended family.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For external validation, we applied our Stanford classifier to unseen data from Geisinger.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“We trained the classifier using 10-fold cross-validation, on the 1:30 prevalence training set, for parameter tuning.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
plainn = 180- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“The MEDPED criteria are based on age-adjusted cholesterol levels and factor into account family pedigree information (if known) with cutoffs designed to identify individuals who would have a causal mutation found on genetic testing.33”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Evaluation via chart review
What the sources say
- Not checked — no full text of the sourcecites [33] Haase, A. & Goldberg, A. C. Identification of people with heterozygous familial hypercholesterolemia. Curr. Opin. Lipidol. 23, 282–289 (2012). source
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“In previous work,40 we built and compared a logistic regression classifier with the random forest classifier.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Not matchedcites [40] Niehaus, K. E., Banda, J. M., Knowles, J. W. & Shah, N. H. FIND FH—A phenotype model to identify patients with familial hypercholesterolemia. in Proceedings of Data Mining for Medical Informatics Workshop 2015 (2015).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and as with most machine learning approaches, having more training data would probably build a better classifier.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
hedgedconditionaln = 197- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
In the paper
“Such upsampling (or downsampling) is a commonly used technique in machine learning with severe class imbalance.39”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
What the sources say
- Not matchedcites [39] Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J. Data Mining: Practical Machine Learning Tools and Techniques. (Morgan Kaufmann, 2016).
Couldn’t be matched to a paper in Crossref, so it can’t be checked.
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
- Not matched
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Before training the classifier, we matched cases and controls by age, gender, and length of record, in order to avoid trivial classification.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Therefore, we used a subset (n = 71) of the 466 Geisinger FH cases and controls (n = 4970) in a 1:70 ratio, to assess the ability of the classifier trained at Stanford to discriminate true cases from non-cases.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
plainn = 71- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“For all patient data, we have complied with all relevant ethical regulations and the study was approved by Stanford University′s institutional review board with waiver of informed patient consent.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“The model building was performed with de-identified data and only the members of the chart review team were provided with access to medical records.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
The paper relies on this without support.
Current guideline-recommended diagnostic strategies for familial hypercholesterolemia largely rely on the availability of untreated LDL-C values and adequate family history information, either or both of which are often unavailable to healthcare providers.
“In addition, this strategy largely relies on the availability of untreated LDL-C values and adequate family history information, either/both of which are often unavailable to the healthcare provider.”
The paper relies on this without support.
The Geisinger familial hypercholesterolemia cases and matched non-cases are correctly labeled as familial hypercholesterolemia and non-familial hypercholesterolemia.
“We validated our classifier and its ability to detect FH patients by evaluating it on an independent dataset from Geisinger”
“466 cases provided by Geisinger”
The paper relies on this without support.
Chart review using DLCN and MEDPED criteria correctly identifies likely familial hypercholesterolemia cases.
“47/56 have a DLCN score of >=3 or are MEDPED positive (84%)”
The paper relies on this without support.
Proven efficacious interventions already exist that can prevent catastrophic cardiovascular events in patients with FH.
“Such case finding is particularly relevant because once a case is found, proven efficacious interventions already exist that can prevent catastrophic cardiovascular events; furthermore, that case can be used to ‘cascade′ to find multiple other cases within an extended family.”
The paper relies on this without support.
A found FH case can be used to cascade to find multiple other cases within an extended family.
“Such case finding is particularly relevant because once a case is found, proven efficacious interventions already exist that can prevent catastrophic cardiovascular events; furthermore, that case can be used to ‘cascade′ to find multiple other cases within an extended family.”
The paper relies on this without support.
The Geisinger Healthcare System data are sufficiently representative of other health systems for the classifier's external validity to generalize.
“We used unseen FH and non-FH patient data from the Geisinger Healthcare System to demonstrate external validity of the classifier.”
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
“However, after applying EHR-based screening, the chance that an individual flagged by our algorithm has FH is 8 out of 10. As a result of this massive chance in post-test prevalence, the cost to find one new case drops to $1429 in genetic counseling and testing”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
Reported 1429 · recomputed 1250
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
“| PPV | 0.88 | 0.85 (0.002) | | Sensitivity | 0.75 | 0.68 (0.002) | | Specificity | 0.99 | 0.99 (0.001) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > Classifier building
“The best performing classifier from this process was tested on the unseen 17 cases and 1190 controls with 1:70 prevalence.”
Rated major: it would touch a result — a value a result rests on, reported and recomputed p-values on opposite sides of the significance level, two passages that conflict with no sign which is right, or a cited work retracted before this paper was published.
In the paper
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Results > Study design
“| PPV | 0.88 | 0.85 (0.002) | | Sensitivity | 0.75 | 0.68 (0.002) | | Specificity | 0.99 | 0.99 (0.001) |”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Methods > External validation in the Geisinger Healthcare System
“we used a subset (n = 71) of the 466 Geisinger FH cases and controls (n = 4970) in a 1:70 ratio”
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
“and 15 min to apply the screening criteria, for each case found we would need to spend roughly $70,000 in genetic testing and 1050 min of clinician time. However, after applying EHR-based screening, the chance that an individual flagged by our algorithm has FH is 8 out of 10. As a result of this massive chance in post-test prevalence, the cost to find one new case drops to $1429 in genetic counseling and testing, and 21.4 min of clinician time.”
§ Finding missed cases of familial hypercholesterolemia in health systems using machine learning > Discussion
Reported 21.4 · recomputed 18.75
Citations
Cites
Breiman, L. (2001) · Machine Learning · doi:10.1023/a:1010933404324 · Cited by 133,538
The Elements of Statistical Learning
Hastie, T., Tibshirani, R., & Friedman, J. (2009) · Springer Series in Statistics · doi:10.1007/978-0-387-84858-7 · Cited by 25,970
Machine learning: Trends, perspectives, and prospects
Jordan, M. I., & Mitchell, T. M. (2015) · Science · doi:10.1126/science.aaa8415 · Cited by 9,955 · cited 1×
“9 Such models have been widely applied in non-medical fi elds 10 with nascent but promising use in medicine.”
MethodStone, N. J., Robinson, J. G., Lichtenstein, A. H., Bairey Merz, C. N., Blum, C. B., Eckel, R. H., Goldberg, A. C., Gordon, D., Levy, D., Lloyd-Jones, D. M., et al. (2014) · Circulation · doi:10.1161/01.cir.0000437738.63853.7a · Cited by 7,081 · cited 1×
“16 The advent of machine learning approaches presents an opportunity to leverage EHR data to develop risk-strati fi cation and predictive models at scale.”
BackgroundStone, N. J., Robinson, J. G., Lichtenstein, A. H., Bairey Merz, C. N., Blum, C. B., Eckel, R. H., Goldberg, A. C., Gordon, D., Levy, D., Lloyd-Jones, D. M., et al. (2014) · Journal of the American College of Cardiology · doi:10.1016/j.jacc.2013.11.002 · Cited by 3,845
Predicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652 · cited 1×
“Machine-learning algorithms can analyze large datasets and determine combinations of variables that consistently classify or predict a certain outcome.(9) Such models have been widely applied in non-medical fields(10) with nascent but promising use in medicine.”
BackgroundDeo, R. C. (2015) · Circulation · doi:10.1161/circulationaha.115.001593 · Cited by 3,466
Nordestgaard, B. G., Chapman, M. J., Humphries, S. E., Ginsberg, H. N., Masana, L., Descamps, O. S., Wiklund, O., Hegele, R. A., Raal, F. J., Defesche, J. C., et al. (2013) · European Heart Journal · doi:10.1093/eurheartj/eht273 · Cited by 2,761 · cited 2×
“1,5,6 Despite the morbidity and mortality associated with FH and the clear bene fi ts of timely management, it is estimated that less than 10% of persons with FH in the US have been diagnosed, 1 with the identi fi cation of index FH cases (probands) as a major bottleneck.”
BackgroundScalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Self-citationScalable and accurate deep learning with electronic health records.
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · eScholarship (California Digital Library) · Cited by 2,242
Self-citation
Research works printed but not linked: 4
- National Collaborating Centre for Primary Care (UK). Clinical guidelines and evidence review for familial hypercholesterolaemia: the identification and management of adults and children with familial hypercholesterolaemia. (Royal College of General Practitioners (UK), 2011).
- Wagstaff, K. Machine Learning that Matters. arXiv [cs.LG] (2012).
- Witten, I. H., Frank, E., Hall, M. A. & Pal, C. J. Data Mining: Practical Machine Learning Tools and Techniques. (Morgan Kaufmann, 2016).
- Niehaus, K. E., Banda, J. M., Knowles, J. W. & Shah, N. H. FIND FH—A phenotype model to identify patients with familial hypercholesterolemia. in Proceedings of Data Mining for Medical Informatics Workshop 2015 (2015).
Cited by
Sorted by citations
Martin, S. S., Aday, A. W., Almarzooq, Z. I., Anderson, C. A. M., Arora, P., Avery, C. L., Baker-Smith, C. M., Barone Gibbs, B., Beaton, A. Z., Boehme, A. K., et al. (2024) · Circulation · doi:10.1161/cir.0000000000001209 · Cited by 2,800
Self-citationMartin, S. S., Aday, A. W., Allen, N. B., Almarzooq, Z., Anderson, C. A. M., Arora, P. K., Avery, C. L., Baker-Smith, C. M., Bansal, N., Beaton, A., et al. (2025) · Circulation · doi:10.1161/cir.0000000000001303 · Cited by 1,284
Self-citationA guide for the diagnosis of rare and undiagnosed disease: beyond the exome
Marwaha, S., Knowles, J. W., & Ashley, E. A. (2022) · Genome Medicine · doi:10.1186/s13073-022-01026-w · Cited by 400
Self-citationPalaniappan, L. P., Allen, N. B., Almarzooq, Z., Anderson, C. A. M., Arora, P. K., Avery, C. L., Baker-Smith, C. M., Bansal, N., Currie, M. E., Earlie, R. S., et al. (2026) · Circulation · doi:10.1161/cir.0000000000001412 · Cited by 343
Self-citationBridging the implementation gap of machine learning in healthcare
Seneviratne, M. G., Shah, N. H., & Chu, L. (2020) · BMJ Innovations · doi:10.1136/bmjinnov-2019-000359 · Cited by 173
“19 The cost utility assessment must take into account the savings (both financial and clinical) associated with early detection, balanced against the cost of a false-positive case being unnecessarily investigated and the costs of deployment and maintenance of the algorithm.”
BackgroundSelf-citation
Metadata
- Authors
- Banda, J. M., Sarraju, A., Abbasi, F., Parizo, J., Pariani, M., Ison, H., Briskin, E., Wand, H., Dubois, S., Jung, K., Myers, S. A., Rader, D. J., Leader, J. B., Murray, M. F., Myers, K. D., Wilemon, K., Shah, N. H., & Knowles, J. W.
- Affiliations
- Georgia State University (US)
- Stanford Cardiovascular Institute (US)
- Stanford University (US)
- FH Foundation (US)
- Geisinger Health System (US)
- Yale University (US)
- Corresponding author
- Shah, Nigam H.
- Knowles, Joshua W.
- ORCID
- Venue
- npj Digital Medicine
- Published
- 2019-04-11
- Volume
- 2
- Issue
- 1
- Pages
- 23-23
- ISSN
- 2398-6352 (ISSN-L)
- PMC
- PMC6550268
- OpenAlex
- W2935996710
- Cited by
- 123
- References
- 38
- Citations by year
- 2026: 7 · 2025: 20 · 2024: 14 · 2023: 18 · 2022: 21 · 2021: 22 · 2020: 14 · 2019: 5
- FWCI
- 7.06
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- npj Digital Medicine (publishedVersion, cc-by)
- PubMed (publishedVersion)
- Europe PMC (PubMed Central) (submittedVersion, cc-by)
- PubMed Central (submittedVersion, cc-by)
- DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion)
- Funders
- American Heart Association (15IRG22930034)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Health Sciences › Health Professions › Health Information Management › Artificial Intelligence in Healthcare
- Health Sciences › Medicine › Cardiology and Cardiovascular Medicine › Lipoproteins and Cardiovascular Health
- Keywords
- familial hypercholesterolemia, missed cases, electronic health records, machine learning classifiers, random forest, underdiagnosis, coronary artery disease risk, external validation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-27, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-27 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
