The number needed to benefit: estimating the value of predictive analytics in healthcare
Open accessThe DOI matches its Crossref record; 17 of the 17 references it cites were found in a registry or an index.
Abstract
Predictive analytics in health care has generated increasing enthusiasm recently, as reflected in a rapidly growing body of predictive models reported in literature and in real-time embedded models using electronic health record data. However, estimating the benefit of applying any single model to a specific clinical problem remains challenging today. Developing a shared framework for estimating model value is therefore critical to facilitate the effective, safe, and sustainable use of predictive tools into the future. We highlight key concepts within the prediction-action dyad that together are expected to impact model benefit. These include factors relevant to model prediction (including the number needed to screen) as well as those relevant to the subsequent action (number needed to treat). In the simplest terms, a number needed to benefit contextualizes the numbers needed to screen and treat, offering an opportunity to estimate the value of a clinical predictive model in action.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Decision curve analysis: a novel method for evaluating prediction models
Vickers, A. J., & Elkin, E. B. (2006) · Medical Decision Making · doi:10.1177/0272989x06295361 · Cited by 6,341
Collins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · BMJ · doi:10.1136/bmj.g7594 · Cited by 4,388 · cited 2×
“9,24,31,32 Together, these elements provide a detailed understanding of a model’s performance with respect to accuracy and applicability across a range of predicted risk thresholds.”
BackgroundPredicting the future - big data, machine learning, and clinical medicine
Obermeyer, Z., & Emanuel, E. J. (2016) · New England Journal of Medicine · doi:10.1056/nejmp1606181 · Cited by 3,652
Scalable and accurate deep learning with electronic health records
Rajkomar, A., Oren, E., Chen, K., Dai, A. M., Hajaj, N., Hardt, M., Liu, P. J., Liu, X., Marcus, J., Sun, M., et al. (2018) · npj Digital Medicine · doi:10.1038/s41746-018-0029-1 · Cited by 2,648
Self-citationAn Assessment of Clinically Useful Measures of the Consequences of Treatment
Laupacis, A., Sackett, D. L., & Roberts, R. S. (1988) · New England Journal of Medicine · doi:10.1056/nejm198806303182605 · Cited by 1,681 · cited 1×
“26,27 30 The product of NNS and NNT or the number needed to benefit (NNB), contextualized with the costs of screening and treatment, can help highlight the costs and benefits of actions that result from responding to a model’s predictions.”
BackgroundAlba, A. C., Agoritsas, T., Walsh, M., Hanna, S., Iorio, A., Devereaux, P. J., McGinn, T., & Guyatt, G. (2017) · JAMA · doi:10.1001/jama.2017.12126 · Cited by 1,631 · cited 1×
“9,24,31,32 Together, these elements provide a detailed understanding of a model’s performance with respect to accuracy and applicability across a range of predicted risk thresholds.”
BackgroundBig Data In Health Care: Using Analytics To Identify And Manage High-Risk And High-Cost Patients
Bates, D. W., Saria, S., Ohno-Machado, L., Shah, A., & Escobar, G. (2014) · Health Affairs · doi:10.1377/hlthaff.2014.0041 · Cited by 1,235
Self-citationMachine Learning and Prediction in Medicine — Beyond the Peak of Inflated Expectations
Chen, J. H., & Asch, S. M. (2017) · New England Journal of Medicine · doi:10.1056/nejmp1702071 · Cited by 1,171
Vickers, A. J., Van Calster, B., & Steyerberg, E. W. (2016) · BMJ · doi:10.1136/bmj.i6 · Cited by 1,163
Goldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
A calibration hierarchy for risk models was defined: from utopia to empirical data
Van Calster, B., Nieboer, D., Vergouwe, Y., De Cock, B., Pencina, M. J., & Steyerberg, E. W. (2016) · Journal of Clinical Epidemiology · doi:10.1016/j.jclinepi.2015.12.005 · Cited by 910 · cited 1×
“9,24,31,32 Together, these elements provide a detailed understanding of a model’s performance with respect to accuracy and applicability across a range of predicted risk thresholds.”
BackgroundGrand challenges in clinical decision support
Sittig, D. F., Wright, A., Osheroff, J. A., Middleton, B., Teich, J. M., Ash, J. S., Campbell, E., & Bates, D. W. (2008) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2007.09.003 · Cited by 582
Self-citationShimabukuro, D. W., Barton, C. W., Feldman, M. D., Mataraso, S. J., & Das, R. (2017) · BMJ Open Respiratory Research · doi:10.1136/bmjresp-2017-000234 · Cited by 382
de Ferranti, S. D., Rodday, A. M., Mendelson, M. M., Wong, J. B., Leslie, L. K., & Sheldrick, R. C. (2016) · Circulation · doi:10.1161/circulationaha.115.018791 · Cited by 380 · cited 1×
“40 Given that over 90% of the U.S. population with FH is undiagnosed, there is a critical need for better screening tools.”
BackgroundCascade Screening for Familial Hypercholesterolemia and the Use of Genetic Testing
Knowles, J. W., Rader, D. J., & Khoury, M. J. (2017) · JAMA · doi:10.1001/jama.2017.8543 · Cited by 190 · cited 1×
“population with FH is undiagnosed, there is a critical need for better screening tools.(41) In a lipid clinic, it is estimated that around 1 in 70 patients with high cholesterol have FH (ie, equivalent to an NNS of 70).”
BackgroundFinding missed cases of familial hypercholesterolemia in health systems using machine learning
Banda, J. M., Sarraju, A., Abbasi, F., Parizo, J., Pariani, M., Ison, H., Briskin, E., Wand, H., Dubois, S., Jung, K., et al. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0101-5 · Cited by 123 · cited 1×
“25).(42) As a result, the diagnostic testing cost to find 1 case would drop to $1250.”
BackgroundSelf-citationDowning, N. L., Rolnick, J., Poole, S. F., Hall, E., Wessels, A. J., Heidenreich, P., & Shieh, L. (2019) · BMJ Quality & Safety · doi:10.1136/bmjqs-2018-008765 · Cited by 112
Cited by
Do no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Self-citationGuo, L. L., Pfohl, S. R., Fries, J., Johnson, A. E. W., Posada, J., Aftandilian, C., Shah, N., & Sung, L. (2022) · Scientific Reports · doi:10.1038/s41598-022-06484-1 · Cited by 103
Self-citationGuo, L. L., Pfohl, S. R., Fries, J., Posada, J., Fleming, S. L., Aftandilian, C., Shah, N., & Sung, L. (2021) · Applied Clinical Informatics · doi:10.1055/s-0041-1735184 · Cited by 96
Self-citationAchieving large-scale clinician adoption of AI-enabled decision support
Scott, I. A., van der Vegt, A., Lane, P., McPhail, S., & Magrabi, F. (2024) · BMJ Health & Care Informatics · doi:10.1136/bmjhci-2023-100971 · Cited by 65
“45 This equation and the estimated costs will vary according to what clinicians perceive as the most clinically appropriate sensitivity and specificity thresholds or cut-off points for the tool which, using simulation methods, determine the net monetary benefit.”
Wang, H. E., Landers, M., Adams, R., Subbaswamy, A., Kharrazi, H., Gaskin, D. J., & Saria, S. (2022) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocac065 · Cited by 63
Metadata
- Authors
- Liu, V. X., Bates, D. W., Wiens, J., & Shah, N. H.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-06-13
- Volume
- 26
- Issue
- 12
- Pages
- 1655-1659
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W2951484576
- Cited by
- 59
- References
- 17
- FWCI
- 7.43
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.