Underserved populations with missing race ethnicity data differ significantly from those with structured race/ethnicity documentation
Open accessAbstract
Objective: We aimed to address deficiencies in structured electronic health record (EHR) data for race and ethnicity by identifying black and Hispanic patients from unstructured clinical notes and assessing differences between patients with or without structured race/ethnicity data. Materials and Methods: Using EHR notes for 16 665 patients with encounters at a primary care practice, we developed rule-based natural language processing (NLP) algorithms to classify patients as black/Hispanic. We evaluated performance of the method against an annotated gold standard, compared race and ethnicity between NLP-derived and structured EHR data, and compared characteristics of patients identified as black or Hispanic using only NLP vs patients identified as such only in structured EHR data. Results: For the sample of 16 665 patients, NLP identified 948 additional patients as black, a 26%increase, and 665 additional patients as Hispanic, a 20% increase. Compared with the patients identified as black or Hispanic in structured EHR data, patients identified as black or Hispanic via NLP only were older, more likely to be male, less likely to have commercial insurance, and more likely to have higher comorbidity. Discussion: Structured EHR data for race and ethnicity are subject to data quality issues. Supplementing structured EHR race data with NLP-derived race and ethnicity may allow researchers to better assess the demographic makeup of populations and draw more accurate conclusions about intergroup differences in health outcomes. Conclusions: Black or Hispanic patients who are not documented as such in structured EHR race/ethnicity fields differ significantly from those who are. Relatively simple NLP can help address this limitation.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Multiple studies recommended comparing and linking data across an individual's record to fill in or correct missing data fields, either to increase completeness and/or to check the accuracy of implausible data.
“This method compares and links data across an individual’s record in order to fill in or correct missing data fields and was recommended by several studies as a way to either increase completeness26,28,49,60,62,64,72 and/or check the accuracy of implausible data.13,26,28”
From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [49]
A study used natural language processing (NLP) to comb through unstructured text fields in clinical notes and used the results to augment race/ethnicity data missing from structured fields, improving identification of race and ethnicity in EHR data.
“Another study used natural language processing (NLP) to “improve the identification of race and ethnicity in EHR data.”49 Those researchers used NLP to comb through the unstructured text fields in clinical notes, then used the results to augment race/ethnicity data missing from structured fields.”
From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [49]
Ten of these (66%) identified that the incomplete data led to validity issues, most commonly that these data were not missing at random and had the potential to introduce bias.
“Ten of these (66%) identified that the incomplete data led to validity issues, most commonly that these data were not missing at random and had the potential to introduced bias.12,34,43–50”
From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [49]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Accuracy of Race, Ethnicity, and Language Preference in an Electronic Health Record
Klinger, E. V., Carlini, S. V., Gonzalez, I., Hubert, S. S., Linder, J. A., Rigotti, N. A., Kontos, E. Z., Park, E. R., Marinacci, L. X., & Haas, J. S. (2015) · Journal of General Internal Medicine · doi:10.1007/s11606-014-3102-8 · Cited by 234 · see citing works
Flagged first, then by citations
Research works printed but not linked: 15
- Nelson A. Unequal treatment: confronting racial and ethnic disparities in health care. J Natl Med Assoc 2002; 94 (8): 666-8.
- Hasnain-Wynia R, Baker DW. Obtaining data on patient race, ethnicity, and primary language in health care organizations: current challenges and proposed solutions. Health Serv Res 2006; 41: 1501-18.
- Hasnain-Wynia R, Van Dyke K, Youdelman M, et al. Barriers to collect- ing patient race, ethnicity, and primary language data in physician practi- ces: an exploratory study. J Natl Med Assoc 2010; 102 (9): 769-75.
- Blumenthal D, Tavenner M. The "meaningful use" regulation for elec- tronic health records. N Engl J Med 2010; 363 (6): 501-4.
- McGarry ME, McColley SA. Minorities are underrepresented in clinical trials of pharmaceutical agents for cystic fibrosis. Ann Am Thorac Soc 2016; 13 (10): 1721-5.
- Johnson SB, Friedman C. Integrating data from natural language process- ing into a clinical information system. Proc AMIA Annu Fall Symp 1996; 1996: 537-41.
- Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language proc- essing: an introduction. J Am Med Inform Assoc 2011; 18 (5): 544-51.
- Heintzelman NH, Taylor RJ, Simonsen L, et al. Longitudinal analysis of pain in patients with metastatic prostate cancer using natural language processing of medical record text. J Am Med Inform Assoc 2013; 20 (5): 898-905.
- Johnson SB, Adekkanattu P, Campion TR, et al. From sour grapes to low- hanging fruit: a case study demonstrating a practical strategy for natural language processing portability. AMIA Jt Summits Transl Sci Proc 2018; 2017: 104-12.
- Sholle ET, Kabariti J, Johnson SB, et al. Secondary use of patients' elec- tronic records (SUPER): an approach for meeting specific data needs of clinical and translational researchers. AMIA Annu Symp Proc 2017; 2017: 1581-8.
- Carrell DS, Cronkite D, Palmer RE, et al. Using natural language process- ing to identify problem usage of prescription opioids. Int J Med Inform 2015; 84 (12): 1057-64.
- Patterson OV, Freiberg MS, Skanderson M, et al. Unlocking echocardio- gram measurements for heart disease research through natural language processing. BMC Cardiovasc Disord 2017; 17: 151.
- Garvin JH, DuVall SL, South BR, et al. Automated extraction of ejection fraction for quality measurement using regular expressions in Unstruc- tured Information Management Architecture (UIMA) for heart failure. J Am Med Inform Assoc 2012; 19 (5): 859-66.
- Adekkanattu P, Sholle ET, DeFerio J, et al. Ascertaining Depression Sever- ity by ExtractingPatient Health Questionnaire-9 (PHQ-9) scores from clinical notes. AMIA Annu Symp Proc 2018; 2018: 147-56.
- Wieland ML, Wu ST, Kaggal VC, et al. Tracking health disparities through natural-language processing. Am J Public Health 2013; 103 (3): 448-9.
Cited by
Sorted by citations
Murphy, M., Scott, L. J., Salisbury, C., Turner, A., Scott, A., Denholm, R., Lewis, R., Iyer, G., Macleod, J., & Horwood, J. (2021) · British Journal of General Practice · doi:10.3399/bjgp.2020.0948 · Cited by 486 · see citing works
Hossain, E., Rana, R., Higgins, N., Soar, J., Barua, P. D., Pisani, A. R., & Turner, K. (2023) · Computers in Biology and Medicine · doi:10.1016/j.compbiomed.2023.106649 · Cited by 407 · see citing works
Electronic Health Record Data Quality Assessment and Tools: A Systematic Review
Lewis, A. E., Weiskopf, N., Abrams, Z. B., Foraker, R., Lai, A. M., Payne, P. R. O., & Gupta, A. (2023) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocad120 · Cited by 171 · see citing works
The quality of social determinants data in the electronic health record: a systematic review
Cook, L. A., Sachs, J., & Weiskopf, N. G. (2021) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocab199 · Cited by 153 · see citing works
- Authors
- Sholle, E. T., Pinheiro, L. C., Adekkanattu, P., Davila, M. A., Johnson, S. B., Pathak, J., Sinha, S., Li, C., Lubansky, S. A., Safford, M. M., & Campion, T. R.
- Affiliations
- Weill Cornell Medicine (US)
- Corresponding author
- Sholle, Evan T
- ORCID
- Venue
- Journal of the American Medical Informatics Association
- Publisher
- Oxford University Press (OUP)
- Published
- 2019-04-26
- Volume
- 26
- Issue
- 8-9
- Pages
- 722-729
- ISSN
- 1067-5027 (ISSN-L)
- 1527-974X
- PMC
- PMC6696506
- OpenAlex
- W2940686655
- Cited by
- 89
- References
- 1
- Citations by year
- 2026: 9 · 2025: 10 · 2024: 9 · 2023: 24 · 2022: 20 · 2021: 7 · 2020: 6 · 2019: 3 · 2018: 1
- FWCI
- 9.80
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model (vor)
- Locations
- Journal of the American Medical Informatics Association (publishedVersion)
- PubMed Central (publishedVersion)
- PubMed (publishedVersion)
- Funders
- Weill Cornell Medicine Clinical and Translational Science Center (UL1 TR000457)
- National Institutes of Health (R01 GM105688, R01 MH105384)
- Topics
- Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
- Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
- Physical Sciences › Computer Science › Artificial Intelligence › Machine Learning in Healthcare
- Keywords
- natural language processing, clinical notes, electronic health records, racial misclassification, Black patients, Hispanic patients, data quality, primary care, underserved populations, health disparities
- MeSH terms
- Algorithms, Black or African American, Cross-Sectional Studies, Ethnicity, Female, Hispanic or Latino, Humans, Male, Natural Language Processing, Vulnerable Populations, Racial Groups, Electronic Health Records
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
