Reporting of demographic data and representativeness in machine learning models using electronic health records
Open accessThe DOI matches its Crossref record; 20 of the 20 references it cites were found in a registry or an index.
Abstract
Objective: The development of machine learning (ML) algorithms to address a variety of issues faced in clinical practice has increased rapidly. However, questions have arisen regarding biases in their development that can affect their applicability in specific populations. We sought to evaluate whether studies developing ML models from electronic health record (EHR) data report sufficient demographic data on the study populations to demonstrate representativeness and reproducibility. Materials and Methods: We searched PubMed for articles applying ML models to improve clinical decision-making using EHR data. We limited our search to papers published between 2015 and 2019. Results: Across the 164 studies reviewed, demographic variables were inconsistently reported and/or included as model inputs. Race/ethnicity was not reported in 64%; gender and age were not reported in 24% and 21% of studies, respectively. Socioeconomic status of the population was not reported in 92% of studies. Studies that mentioned these variables often did not report if they were included as model inputs. Few models (12%) were validated using external populations. Few studies (17%) open-sourced their code. Populations in the ML studies include higher proportions of White and Black yet fewer Hispanic subjects compared to the general US population. Discussion: The demographic characteristics of study populations are poorly reported in the ML literature based on EHR data. Demographic representativeness in training data and model transparency is necessary to ensure that ML models are deployed in an equitable and reproducible manner. Wider adoption of reporting guidelines is warranted to improve representativeness and reproducibility.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 2×
“When demographic information was reported, we observed trends similar to previous analyses, 23 with an unbalanced distribution of race/ethnicity.”
MethodBenchimol, E. I., Smeeth, L., Guttmann, A., Harron, K., Moher, D., Petersen, I., Sørensen, H. T., von Elm, E., Langan, S. M., & RECORD Working Committee (2015) · PLOS Medicine · doi:10.1371/journal.pmed.1001885 · Cited by 5,749
A manifesto for reproducible science
Munafò, M. R., Nosek, B. A., Bishop, D. V. M., Button, K. S., Chambers, C. D., Percie du Sert, N., Simonsohn, U., Wagenmakers, E.-J., Ware, J. J., & Ioannidis, J. P. A. (2017) · Nature Human Behaviour · doi:10.1038/s41562-016-0021 · Cited by 3,397
Self-citationInternal validation of predictive models
Steyerberg, E. W., Harrell, F. E., Borsboom, G. J. J. M., Eijkemans, M. J. C., Vergouwe, Y., & Habbema, J. D. F. (2001) · Journal of Clinical Epidemiology · doi:10.1016/s0895-4356(01)00341-9 · Cited by 2,746
Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning
Poplin, R., Varadarajan, A. V., Blumer, K., Liu, Y., McConnell, M. V., Corrado, G. S., Peng, L., & Webster, D. R. (2018) · Nature Biomedical Engineering · doi:10.1038/s41551-018-0195-0 · Cited by 1,865
Implementing Machine Learning in Health Care — Addressing Ethical Challenges
Char, D. S., Shah, N. H., & Magnus, D. (2018) · New England Journal of Medicine · doi:10.1056/nejmp1714229 · Cited by 1,857 · cited 1×
“7 Repre-sentativeness may be a larger problem in countries with significant health disparities on the basis of demographics, such as the United States (US).”
BackgroundMoons, K. G. M., Wolff, R. F., Riley, R. D., Whiting, P. F., Westwood, M., Collins, G. S., Reitsma, J. B., Kleijnen, J., & Mallett, S. (2019) · Annals of Internal Medicine · doi:10.7326/m18-1377 · Cited by 1,669
George, H., Duke Jon, D., Shah Nigam, H., Reich Christian, G., Vojtech, H., Schuemie Martijn, J., Suchard Marc, A., Woong, P. R., Kei, W. I. C., Rijnbeek Peter, R., et al. (2015) · Studies in Health Technology and Informatics · doi:10.3233/978-1-61499-564-7-574 · Cited by 1,542 · cited 1×
“23,30 To facilitate broader dissemination of ML technologies, published models should strive to provide their source code where possible.”
Rajpurkar, P., Irvin, J., Ball, R. L., Zhu, K., Yang, B., Mehta, H., Duan, T., Ding, D., Bagul, A., Langlotz, C. P., et al. (2018) · PLOS Medicine · doi:10.1371/journal.pmed.1002686 · Cited by 1,449
Potential Biases in Machine Learning Algorithms Using Electronic Health Record Data
Gianfrancesco, M. A., Tamang, S., Yazdany, J., & Schmajuk, G. (2018) · JAMA Internal Medicine · doi:10.1001/jamainternmed.2018.3763 · Cited by 1,395 · cited 1×
“5,6 As ML tools proliferate across clinical settings, it is important to understand potential demographic biases underlying model development.”
BackgroundDo no harm: a roadmap for responsible machine learning for health care
Wiens, J., Saria, S., Sendak, M., Ghassemi, M., Liu, V. X., Doshi-Velez, F., Jung, K., Heller, K., Kale, D., Saeed, M., et al. (2019) · Nature Medicine · doi:10.1038/s41591-019-0548-6 · Cited by 1,266
Nagendran, M., Chen, Y., Lovejoy, C. A., Gordon, A. C., Komorowski, M., Harvey, H., Topol, E. J., Ioannidis, J. P. A., Collins, G. S., & Maruthappu, M. (2020) · BMJ · doi:10.1136/bmj.m689 · Cited by 1,152
Self-citationGoldstein, B. A., Navar, A. M., Pencina, M. J., & Ioannidis, J. P. A. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw042 · Cited by 936
Self-citationKennedy-Martin, T., Curtis, S., Faries, D., Robinson, S., & Johnston, J. (2015) · Trials · doi:10.1186/s13063-015-1023-4 · Cited by 878 · cited 1×
“25 However, this approach also requires disclosure of demographic data and inclusion/exclusion criteria.”
BackgroundMachine Learning and Health Care Disparities in Dermatology
Adamson, A. S., & Smith, A. (2018) · JAMA Dermatology · doi:10.1001/jamadermatol.2018.2348 · Cited by 639 · cited 1×
“8,9 13 This may be further exacerbated by the lack of clarity related to model development and evaluation.”
BackgroundRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · cited 2×
“21,28 However, many of the studies screened exclusively used data from a single facility, and the models were rarely validated in external populations.”
BackgroundHersh, W. R., Weiner, M. G., Embi, P. J., Logan, J. R., Payne, P. R. O., Bernstam, E. V., Lehmann, H. P., Hripcsak, G., Hartzog, T. H., Cimino, J. J., et al. (2013) · Medical Care · doi:10.1097/mlr.0b013e31829b1dbd · Cited by 567 · cited 1×
“27 Clearly, many sensitive variables, such as socioeconomic or housing status may not be available in EHRs, hence the low reporting we found on these variables is not surprising.”
BackgroundPathologist-level interpretable whole-slide cancer diagnosis with deep learning
Zhang, Z., Chen, P., McGough, M., Xing, F., Wang, C., Bui, M., Xie, Y., Sapkota, M., Cui, L., Dhillon, J., et al. (2019) · Nature Machine Intelligence · doi:10.1038/s42256-019-0052-1 · Cited by 311 · cited 2×
“Recently, reports have questioned whether ML models in health-care might perpetuate discrimination if trained on historical data— which is often poorly representative of broader populations.”
BackgroundPresenting machine learning model information to clinical end users with model facts labels
Sendak, M. P., Gao, M., Brajer, N., & Balu, S. (2020) · npj Digital Medicine · doi:10.1038/s41746-020-0253-3 · Cited by 279 · cited 1×
“21,24 To assess population representativeness of ML models’ training population, direct access to study data is often necessary, as is some-times done for randomized controlled trials.”
BackgroundPutting the data before the algorithm in big data addressing personalized healthcare
Cahan, E. M., Hernandez-Boussard, T., Thadaney-Israni, S., & Rubin, D. L. (2019) · npj Digital Medicine · doi:10.1038/s41746-019-0157-2 · Cited by 149 · cited 1×
“Gender 124 (76) 30 (18) 94 (57) 40 (24) 20 (12) 20 (12) Age 129 (79) 16 (10) 113 (69) 35 (21) 12 (7) 23 (14) Race/ethnicity 59 (36) 18 (11) 41 (25) 105 (64) 86 (52) 19 (12) Socio-economic status 14 (8) 1 (1) 13 (8) 150 (92) 145 (88) 5 (3) Other sensitive demographic variables Marital status 9 (6) – 9 (6) 155 (95) 149…”
Self-citation
Cited by
Hernandez-Boussard, T., Bozkurt, S., Ioannidis, J. P. A., & Shah, N. H. (2020) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocaa088 · Cited by 410
Self-citationDigital twins for predictive oncology will be a paradigm shift for precision cancer care
Hernandez-Boussard, T., Macklin, P., Greenspan, E. J., Gryshuk, A. L., Stahlberg, E., Syeda-Mahmood, T., & Shmulevich, I. (2021) · Nature Medicine · doi:10.1038/s41591-021-01558-5 · Cited by 219
Self-citationEvaluation and Mitigation of Racial Bias in Clinical Machine Learning Models: Scoping Review
Huang, J., Galal, G., Etemadi, M., & Vaidyanathan, M. (2022) · JMIR Medical Informatics · doi:10.2196/36388 · Cited by 186
“Moreover, reporting of demographic information needed to assess biases due to data sets is lacking [62,63].”
Busch, F., Adams, L. C., & Bressem, K. K. (2023) · Medical Science Educator · doi:10.1007/s40670-023-01815-x · Cited by 88
“In terms of diversity of training data, institutions need to ensure that developers train their algorithms and models on high-quality, diverse data that accurately represent the population being studied so that the resulting educational content is unbiased and fair [38, 39].”
BackgroundThe AI life cycle: a holistic approach to creating ethical AI for health decisions
Ng, M. Y., Kapur, S., Blizinsky, K. D., & Hernandez-Boussard, T. (2022) · Nature Medicine · doi:10.1038/s41591-022-01993-y · Cited by 71
Self-citation
Metadata
- Authors
- Bozkurt, S., Cahan, E. M., Seneviratne, M. G., Sun, R., Lossio-Ventura, J. A., Ioannidis, J. P. A., & Hernandez-Boussard, T.
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2020-09-16
- Volume
- 27
- Issue
- 12
- Pages
- 1878-1884
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- OpenAlex
- W3087629119
- Cited by
- 65
- References
- 20
- FWCI
- 2.37
- Type
- journal article
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.