CoChat
Sign Up
Cited by 57Medical Coding and Health InformationElectronic Health Records SystemsGlobal Cancer Incidence and Screening

Assessing race and ethnicity data quality across cancer registries and EMRs in two hospitals

Open access
Lee, Simon J Craddock, Grobe, James E, Tiro, Jasmin A·2016·Journal of the American Medical Informatics Association
DOI 10.1093/jamia/ocv156PMC PMC6095103
Open this paper in CoChat — free account

Abstract

Background Measurement of patient race/ethnicity in electronic health records is mandated and important for tracking health disparities. Objective Characterize the quality of race/ethnicity data collection efforts. Methods For all cancer patients diagnosed (2007–2010) at two hospitals, we extracted demographic data from five sources: 1) a university hospital cancer registry, 2) a university electronic medical record (EMR), 3) a community hospital cancer registry, 4) a community EMR, and 5) a joint clinical research registry. The patients whose data we examined (N = 17 834) contributed 41 025 entries (range: 2–5 per patient across sources), and the source comparisons generated 1–10 unique pairs per patient. We used generalized estimating equations, chi-squares tests, and kappas estimates to assess data availability and agreement. Results Compared to sex and insurance status, race/ethnicity information was significantly less likely to be available (χ2 > 8043, P < .001), with variation across sources (χ2 > 10 589, P < .001). The university EMR had a high prevalence of “Unknown” values. Aggregate kappa estimates across the sources was 0.45 (95% confidence interval, 0.45–0.45; N = 31 276 unique pairs), but improved in sensitivity analyses that excluded the university EMR source (κ = 0.89). Race/ethnicity data were in complete agreement for only 6988 patients (39.2%). Pairs with a “Black” data value in one of the sources had the highest agreement (95.3%), whereas pairs with an “Other” value exhibited the lowest agreement across sources (11.1%). Discussion Our findings suggest that high-quality race/ethnicity data are attainable. Many of the “errors” in race/ethnicity data are caused by missing or “Unknown” data values. Conclusions To facilitate transparent reporting of healthcare delivery outcomes by race/ethnicity, healthcare systems need to monitor and enforce race/ethnicity data collection standards.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1093/jamia/ocv156

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Fourteen studies reported that Hispanic patients were more likely to be misclassified, either that information about their ethnicity was missing or they had been mistakenly grouped into the "Other" category.

    “Fourteen studies reported that Hispanic patients were more likely to be misclassified, either that information about their ethnicity was missing or they had been mistakenly grouped into the “Other” category.11,23–25,29,30,32,33,36–41”

    From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [23]

    The Discussion reports mismatches between “White” and “Hispanic” and poor-quality records coded “Other” or “Unknown.” It does not establish that Hispanic patients were more likely than other patients to have missing ethnicity or to be placed in “Other.” The count of fourteen studies belongs to the citing paper’s collective statement, not to this individual source. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

  • Three studies in the review found that casewise deletion decreased the quality of race/ethnicity data.

    “However, 3 studies in our review found that casewise deletion decreased the quality of race/ethnicity data.23,50,65”

    From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [23]

    The Concordance section (page unavailable) reports that excluding records with 'Unknown' race/ethnicity values eliminated over 44% of the pairs, supporting a loss of information from deletion. But the Discussion says the exclusion substantially increased agreement; the paper does not conclude that deletion decreased data quality overall. The count of three studies belongs to the citing review, not to this individual source. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Misclassification bias was noted as a problem or potential problem in 18 of the 26 (69%) articles about the plausibility of race/ethnicity data.

    “Misclassification bias, that is, incorrect assignment, was noted as a problem or a potential problem in 18 of the 26 (69%) articles about the plausibility of race/ethnicity data.11,13,23–38”

    From The quality of social determinants data in the electronic health record: a systematic review (2021) · cites this paper as [23]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 30

  1. [1]1. Thorlby R, Jorgensen S, Siegel B, Ayanian JZ. How health care organizations are using data on patients' race and ethnicity to improve quality of care. Milbank Q. 2011;89(2):226-255.
  2. [2]2. Mays VM, Ponce NA, Washington DL, Cochran SD. Classification of race and ethnicity: Implications for public health. Annu Rev Public Health. 2003;24(1):83-110.
  3. [3]3. Hahn RA. The state of federal health statistics on racial and ethnic groups. JAMA. 1992;267(2):268-271.
  4. [4]4. Manly JJ. Deconstructing race and ethnicity - Implications for measurement of health outcomes. Medical Care. 2006;44(11):S10-S16.
  5. [6]6. Clegg LX, Reichman ME, Hankey BF et al . Quality of race, Hispanic ethnicity, and immigrant status in population-based cancer registry data: implications for health disparity studies. Cancer Causes Control. 2007; 18(2):177-187.
  6. [8]8. Wynia MK, Ivey SL, Hasnain-Wynia R. Collection of data on patients' race and ethnic group by physician practices. N Engl J Med. 2010;362(9):846-850.
  7. [9]9. Nerenz DR, Carreon R, Veselovskiy G. Race, ethnicity, and language data collection by health plans: findings from 2010 AHIPF-RWJF survey. J Health Care Poor Underserved. 2013;24(4):1769-1783.
  8. [10]10. Fiellin M, Chemerynski S, Borak J. Race, ethnicity, and the Seer database. Med Pediatr Oncol. 2003;41(5):413-414.
  9. [11]11. Blustein J. The reliability of racial classifications in hospital discharge abstract data. Am J Public Health. 1994;84(6):1018-1021.
  10. [12]12. Schrag D. Enhancing cancer registry data to promote rational health system design. J Natl Cancer Inst. 2008;100(6):378-379.
  11. [13]13. Johnson PJ, Blewett LA, Davern M. Disparities in public use data availability for race, ethnic, and immigrant groups national surveys for healthcare disparities research. Medical Care. 2010;48(12):1122-1127.
  12. [14]14. Shea CM, Malone R, Weinberger M et al . Assessing organizational capacity for achieving meaningful use of electronic health records. Health Care Manage Rev. 2014;39(2):124-133.
  13. [15]15. Rubin DB, Singh SR, Jacobson PD. Evaluating hospitals' provision of community benefit: an argument for an outcome-based approach to nonprofit hospital tax exemption. Am J Public Health. 2013;103(4):612-616.
  14. [16]16. Gomez SL, Le GM, West DW, Satariano WA, O'Connor L. Hospital policy and practice regarding the collection of data on race, ethnicity, and birthplace. Am J Public Health. 2003;93(10):1685-1688.
  15. [18]18. Ramirez AG, Miller AR, Gallion K, San Miguel de MS, Chalela P, Garcia AS. Testing three different cancer genetics registry recruitment methods with Hispanic cancer patients and their family members previously registered in local cancer registries in Texas. Community Genet. 2008;11(4):215-223.
  16. [19]19. Gomez SL, Glaser SL. Misclassification of race/ethnicity in a populationbased cancer registry (United States). Cancer Causes Control. 2006;17(6): 771-781.
  17. [23]23. Koering SM. The role of the cancer registrar. How cancer registrars ensure quality data for public reporting. J AHIMA. 2008;79(3):66-67.
  18. [24]24. Lee SJC. Organizational practice and social constraints: Problems of racial identity data collectionin cancer care and research. In: Laura EG, Nancy L, eds. Mapping 'race": Critical Approaches to Health Disparities Research . New Brunswich: Rutgers University Press; 2013: 87-103.
  19. [25]25. Wallman KK, Evinger S, Schechter S. Measuring our nation's diversity: developing a common language for data on race/ethnicity. Am J Public Health. 2000;90(11):1704-1708.
  20. [27]27. Lin L, Hedayat AS, Wu W. A unified approach for assessing agreement for continuous and categorical data. J Biopharm Stat. 2007;17(4):629-652.
  21. [28]28. Vanbelle S, Mutsvari T, Declerck D, Lesaffre E. Hierarchical modeling of agreement. Stat Med. 2012;31(28):3667-3680.
  22. [29]29. Ruddat I, Scholz B, Bergmann S et al . Statistical tools to improve assessing agreement between several observers. Animal . 2014;8(4):643-649.
  23. [30]30. Stewart SL, Swallen KC, Glaser SL, Horn-Ross PL, West DW. Comparison of methods for classifying Hispanic ethnicity in a population-based cancer registry. Am J Epidemiol. 1999;149(11):1063-1071.
  24. [31]31. Uebersax JS. A design-independent method for measuring the reliability of psychiatric diagnosis. J Psychiatr Res. 1982;17(4):335-342.
  25. [32]32. Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43(6):551-558.
  26. [33]33. Lin SS, O'Malley CD, Lui SW. Factors associated with missing birthplace information in a population-based cancer registry. Ethn Dis. 2001;11(4):598-605.
  27. [34]34. Laws MB, Heckscher RA. Racial and ethnic identification practices in public health data systems in New England. Public Health Rep. 2002;117(1):50-61.
  28. [35]35. Jorgensen S, Thorlby R, Weinick RM, Ayanian JZ. Responses of Massachusetts hospitals to a state mandate to collect race, ethnicity and language data from patients: a qualitative study. BMC Health Serv Res. 2010;10:352.
  29. [36]36. Baker DW, Hasnain-Wynia R, Kandula NR, Thompson JA, Brown ER. Attitudes toward health care providers, collecting information about patients' race, ethnicity, and language. Med Care. 2007;45(11):1034-1042.
  30. [37]37. Ulmer C, McFadden B, Nerenz DR. Race, Ethnicity, and Language Data:: Standardization for Health Care Quality Improvement . Washington DC: National Academies Press; 2009.

Cited by

Sorted by citations

Authors
Lee, S. J. C., Grobe, J. E., & Tiro, J. A.
Affiliations
  • Harold C. Simmons Comprehensive Cancer Center (US)
  • The University of Texas Southwestern Medical Center (US)
Corresponding author
Lee, Simon J Craddock
ORCID
Venue
Journal of the American Medical Informatics Association
Publisher
Oxford University Press (OUP)
Published
2015-12-11
Volume
23
Issue
3
Pages
627-634
ISSN
  • 1067-5027 (ISSN-L)
  • 1527-974X
DOI
10.1093/jamia/ocv156
PMC
PMC6095103
OpenAlex
W2194109005
Cited by
57
References
3
Citations by year
2026: 3 · 2025: 7 · 2024: 3 · 2023: 16 · 2022: 8 · 2021: 7 · 2020: 3 · 2019: 5 · 2018: 1 · 2017: 3 · 2016: 1
FWCI
3.21
Type
journal article
Language
en
Open access
Locations
  • Journal of the American Medical Informatics Association (publishedVersion)
  • Europe PMC (PubMed Central) (publishedVersion)
  • PubMed Central (publishedVersion)
  • PubMed (publishedVersion)
  • HighWire Press Open Archive (submittedVersion)
Funders
NIH (UL1TR001105)
Topics
  • Health Sciences › Health Professions › Health Information Management › Medical Coding and Health Information
  • Health Sciences › Health Professions › Health Information Management › Electronic Health Records Systems
  • Health Sciences › Medicine › Oncology › Global Cancer Incidence and Screening
Keywords
cancer registry, electronic health records, health disparities, data source agreement, missing data, data collection standards, cancer patients
MeSH terms
Data Accuracy, Adolescent, Adult, Aged, Aged, 80 and over, Ethnicity, Female, Hospitals, Community, Hospitals, University, Humans, Male, Middle Aged, Neoplasms, Registries, United States, Racial Groups, Young Adult, Electronic Health Records

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review