Leveraging Administrative Data for Bias Audits: Assessing Disparate Coverage with Mobility Data for COVID-19 Policy
Open accessAbstract
Anonymized smartphone-based mobility data has been widely adopted in devising and evaluating COVID-19 response strategies such as the targeting of public health resources. Yet little attention has been paid to measurement validity and demographic bias, due in part to the lack of documentation about which users are represented as well as the challenge of obtaining ground truth data on unique visits and demographics. We illustrate how linking large-scale administrative data can enable auditing mobility data for bias in the absence of demographic information and ground truth labels. More precisely, we show that linking voter roll data---containing individual-level voter turnout for specific voting locations along with race and age---can facilitate the construction of rigorous bias and reliability tests. Using data from North Carolina's 2018 general election, these tests illuminate a sampling bias that is particularly noteworthy in the pandemic context: older and non-white voters are less likely to be captured by mobility data. We show that allocating public health resources based on such mobility data could disproportionately harm high-risk elderly and minority groups.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Auditor tenure is a key determinant of independence.
“The tenure of auditors is another key determinant of independence [15].”
From Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance (2022) · cites this paper as [15]
Section 4 describes an audit of demographic disparities in mobility-data coverage, and §4.5 applies it to voter turnout. Searches for auditor tenure, appointment length, and independence found no passage addressing whether tenure determines auditor independence. The source therefore does not verify the cited claim. (LLM, conf=0.78)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Zhou, F., Yu, T., Du, R., Fan, G., Liu, Y., Liu, Z., Xiang, J., Wang, Y., Song, B., Gu, X., et al. (2020) · The Lancet · doi:10.1016/s0140-6736(20)30566-3 · Cited by 27,263 · see citing works
Crenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510 · see citing works
Dissecting racial bias in an algorithm used to manage the health of populations
Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019) · Science · doi:10.1126/science.aax2342 · Cited by 7,137 · cited 1× · see citing works
“Important work has audited demographic bias of facial recognition technology [8], child welfare screening tools [11], criminal risk assessment scores [43], and health care allocation tools [2, 39].”
BackgroundStatistics and Causal Inference
Holland, P. W. (1986) · Journal of the American Statistical Association · doi:10.1080/01621459.1986.10478354 · Cited by 5,293 · see citing works
The Parable of Google Flu: Traps in Big Data Analysis
Lazer, D., Kennedy, R., King, G., & Vespignani, A. (2014) · Science · doi:10.1126/science.1248506 · Cited by 2,523 · cited 1× · see citing works
“Ground truth is often lacking, which is precisely why policymakers and academics have flocked toward big data, on the implicit assumption that scale can overcome more conventional questions of data reliability, sampling bias, and the like [2, 35].”
BackgroundHidden in Plain Sight — Reconsidering the Use of Race Correction in Clinical Algorithms
Vyas, D. A., Eisenstein, L. G., & Jones, D. S. (2020) · New England Journal of Medicine · doi:10.1056/nejmms2004740 · Cited by 1,731 · see citing works
Mobility network models of COVID-19 explain inequities and inform reopening
Chang, S., Pierson, E., Koh, P. W., Gerardin, J., Redbird, B., Grusky, D., & Leskovec, J. (2021) · Nature · doi:10.1038/s41586-020-2923-3 · Cited by 1,533 · see citing works
The Disproportionate Impact of COVID-19 on Racial and Ethnic Minorities in the United States
Tai, D. B. G., Shah, A., Doubeni, C. A., Sia, I. G., & Wieland, M. L. (2020) · Clinical Infectious Diseases · doi:10.1093/cid/ciaa815 · Cited by 1,390 · see citing works
Allcott, H., Boxell, L., Conway, J., Gentzkow, M., Thaler, M., & Yang, D. (2020) · Journal of Public Economics · doi:10.1016/j.jpubeco.2020.104254 · Cited by 999 · see citing works
Moreland, A., Herlihy, C., Tynan, M. A., Sunshine, G., McCord, R. F., Hilton, C., Poovey, J., Werner, A. K., Jones, C. D., Fulmer, E. B., et al. (2020) · MMWR. Morbidity and Mortality Weekly Report · doi:10.15585/mmwr.mm6935a2 · Cited by 614 · see citing works
A comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601 · see citing works
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444 · see citing works
Gender Differences in Privacy-Related Measures for Young Adult Facebook Users
Hoy, M. G., & Milne, G. (2010) · Journal of Interactive Advertising · doi:10.1080/15252019.2010.10722168 · Cited by 443 · see citing works
The use of mobile phone data to inform analysis of COVID-19 pandemic epidemiology
Grantz, K. H., Meredith, H. R., Cummings, D. A. T., Metcalf, C. J. E., Grenfell, B. T., Giles, J. R., Mehta, S. H., Solomon, S. S., Labrique, A. B., Kishore, N., et al. (2020) · Nature Communications · doi:10.1038/s41467-020-18190-5 · Cited by 404 · see citing works
Shavers, V. L., Lynch, C. F., Burmeister, L. F., & Torner, J. C. (1997) · Ethnicity and Health · doi:10.1080/13557858.1997.9961813 · Cited by 376 · see citing works
The Frontiers of Fairness in Machine LearningPreprint
Chouldechova, A., & Roth, A. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1810.08810 · Cited by 372 · see citing works
Self-citationBernal, G., & Río, M. R. S. (2001) · Cultural Diversity & Ethnic Minority Psychology · doi:10.1037/1099-9809.7.4.328 · Cited by 353 · see citing works
Oneto, L., & Chiappa, S. (2020) · Studies in computational intelligence · doi:10.1007/978-3-030-43883-8_7 · Cited by 315 · see citing works
(2018) · Cited by 230 · see citing works
Self-citationMapping county-level mobility pattern changes in the United States in response to COVID-19
Gao, S., Rao, J., Kang, Y., Liang, Y., & Kruse, J. (2020) · SIGSPATIAL Special · doi:10.1145/3404820.3404824 · Cited by 226 · see citing works
How age and gender affect smartphone usage
Andone, I., Błaszkiewicz, K., Eibes, M., Trendafilov, B., Montag, C., & Markowetz, A. (2016) · Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct · doi:10.1145/2968219.2971451 · Cited by 220 · see citing works
Risk, Race, and Recidivism: Predictive Bias and Disparate Impact
Skeem, J., & Lowenkamp, C. T. (2016) · Criminology · doi:10.1111/1745-9125.12123 · Cited by 217 · see citing works
Heterogeneous Mobile Phone Ownership and Usage Patterns in Kenya
Wesolowski, A., Eagle, N., Noor, A. M., Snow, R. W., & Buckee, C. O. (2012) · PLoS ONE · doi:10.1371/journal.pone.0035319 · Cited by 216 · see citing works
Wanted Single, White Male for Medical Research
Dresser, R. S. (1992) · The Hastings Center Report · doi:10.2307/3562720 · Cited by 215 · see citing works
Connecting Mobility to Infectious Diseases: The Promise and Limits of Mobile Phone Data
Wesolowski, A., Buckee, C. O., Engø‐Monsen, K., & Metcalf, C. J. E. (2016) · The Journal of Infectious Diseases · doi:10.1093/infdis/jiw273 · Cited by 212 · see citing works
Ethnic Minority Older Adults Participating in Clinical Research
Moreno-John, G., Gachie, A., Fleming, C. M., Nápoles‐Springer, A. M., Mutran, E. J., Manson, S. M., & Pérez‐Stable, E. J. (2004) · Journal of Aging and Health · doi:10.1177/0898264304268151 · Cited by 205 · see citing works
Assessing algorithmic fairness with unobserved protected class using data combination
Kallus, N., Mao, X., & Zhou, A. (2019) · FAT* · doi:10.1145/3351095.3373154 · Cited by 203 · see citing works
COVID-19 and the other pandemic: populations made vulnerable by systemic inequity
Gray, D. M., Anyane‐Yeboa, A., Balzora, S., Issaka, R. B., & May, F. P. (2020) · Nature Reviews Gastroenterology & Hepatology · doi:10.1038/s41575-020-0330-8 · Cited by 202 · see citing works
Benzell, S. G., Collis, A., & Nicolaides, C. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2008025117 · Cited by 149 · see citing works
Randomization Inference With Natural Experiments
Ho, D. E., & Imai, K. (2006) · Journal of the American Statistical Association · doi:10.1198/016214505000001258 · Cited by 142 · see citing works
Self-citationMeasures of Human Mobility Using Mobile Phone Records Enhanced with GIS Data
Williams, N. E., Thomas, T. A., Dunbar, M. D., Eagle, N., & Dobra, A. (2015) · PLoS ONE · doi:10.1371/journal.pone.0133630 · Cited by 112 · see citing works
Mapping county-level mobility pattern changes in the United States in response to COVID-19
Gao, S., Rao, J., Kang, Y., Liang, Y., & Kruse, J. (2020) · SIGSPATIAL Special · doi:10.1145/3404111.3404115 · Cited by 108 · see citing works
A County-level Dataset for Informing the United States' Response to COVID-19Preprint
Killeen, B. D., Wu, J. Y., Shah, K., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.00756 · Cited by 88 · see citing works
Auditing Algorithms for Discrimination
Kim, P. T. (2017) · Open Scholarship Institutional Repository (Washington University in St. Louis) · Cited by 77 · see citing works
Lee, S.-H., Brick, J. M., Brown, E. R., & Grant, D. (2010) · Health Services Research · doi:10.1111/j.1475-6773.2010.01120.x · Cited by 74 · see citing works
Requiring smartphone ownership for mHealth interventions: who could be left out?
Bommakanti, K. K., Smith, L. L., Liu, L., Do, D., Cuevas-Mota, J., Collins, K., Munoz, F., Rodwell, T. C., & Garfein, R. S. (2020) · BMC Public Health · doi:10.1186/s12889-019-7892-9 · Cited by 66 · see citing works
Minorities, Women, and Clinical Cancer Research
Underwood, S. M. (2000) · Annals of Epidemiology · doi:10.1016/s1047-2797(00)00200-3 · Cited by 51 · see citing works
The Cost of Staying Open: Voluntary Social Distancing and Lockdowns in the USPreprint
Brzezinski, A., Kecht, V., & Dijcke, D. V. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3614494 · Cited by 39 · see citing works
Bao, H., Zhou, X. S., Zhang, Y., Li, Y., & Xie, Y. (2020) · ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems (SIGSPATIAL GIS) · doi:10.1145/3397536.3422261 · Cited by 34 · see citing works
The Contagion Externality of a Superspreading Event: The Sturgis Motorcycle Rally and COVID-19
Dave, D., Friedson, A., McNichols, D., & Sabia, J. J. (2020) · National Bureau of Economic Research · doi:10.3386/w27813 · Cited by 34 · see citing works
Internal and External Effects of Social Distancing in a PandemicPreprint
Farboodi, M., Jarosch, G., & Shimer, R. J. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3579559 · Cited by 34 · see citing works
Chang, S. Y., Pierson, E., Koh, P. W., et al. (2020) · medRxiv · doi:10.1101/2020.06.15.20131979 · Cited by 24 · see citing works
Bias From Wireless Substitution in Surveys of Hispanics
Dutwin, D., Keeter, S., & Kennedy, C. (2010) · Hispanic Journal of Behavioral Sciences · doi:10.1177/0739986310365479 · Cited by 14 · see citing works
FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning
Cabrera, Á. A., Epperson, W., Hohman, F., Kahng, M., Morgenstern, J., & Chau, D. H. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986948 · Cited by 14 · see citing works
Divided We Stay Home: Social Distancing and Ethnic Diversity
Egorov, G., Сергеевич, Е. Р., Makarin, A., & Petrova, M. (2020) · National Bureau of Economic Research · doi:10.3386/w27277 · Cited by 14 · see citing works
Altenburger, K. M., & Ho, D. E. (2018) · Journal of Institutional and Theoretical Economics JITE · doi:10.1628/jite-2019-0001 · Cited by 13 · see citing works
Self-citation
Flagged first, then by citations
1 of the 47 references couldn’t be matched to a known paper.
Research works printed but not linked: 2
- Han Bao, Xun Zhou, Yingxue Zhang, Yanhua Li, and Yiqun Xie. 2020. COVID- GAN: Estimating Human Mobility Responses to COVID-19 Pandemic through Spatio-Temporal Conditional Generative Adversarial Networks. In Proceedings of the 28th International Conference on Advances in Geographic Information Systems. 273-282.
- Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accu- racy Disparities in Commercial Gender Classication. In Conference on Fairness, Accountability, and Transparency (Proceedings of Machine Learning Research),
Cited by
Sorted by citations
Understanding socioeconomic disparities in travel behavior during the COVID‐19 pandemic
Brough, R., Freedman, M. L., & Phillips, D. C. (2021) · Journal of Regional Science · doi:10.1111/jors.12527 · Cited by 180 · see citing works
Auditing of AI: Legal, Ethical and Technical Approaches
Mökander, J. (2023) · Digital Society · doi:10.1007/s44206-023-00074-y · Cited by 152 · see citing works
“…image classification systems (Morina et al., 2019), for accuracy in news curation systems (Bandy & Diakopoulos, 2019), for completeness in datasets (Coston et al., 2021; Sookhak et al., 2014), and for data privacy, e.g., how easy it is to reconstruct training data from AI systems (Kolhar et al.,…”
BackgroundLi, Z., Ning, H., Jing, F., & Lessani, M. N. (2024) · PLoS ONE · doi:10.1371/journal.pone.0294430 · Cited by 124 · see citing works
Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governance
Raji, I. D., Xu, P., Honigsberg, C., & Ho, D. (2022) · Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3514094.3534181 · Cited by 122 · see citing works
Self-citationBelief in Science Influences Physical Distancing in Response to COVID-19 Lockdown PoliciesPreprint
Brzezinski, A., Kecht, V., Dijcke, D. V., & Wright, A. L. (2020) · SSRN Electronic Journal · doi:10.2139/ssrn.3587990 · Cited by 115 · see citing works
Larson, L. R., Zhang, Z., Oh, J. I., Beam, W., Ogletree, S. S., Bocarro, J. N., Lee, K. J., Casper, J. M., Stevenson, K. T., Hipp, J. A., et al. (2021) · Frontiers in Sustainable Cities · doi:10.3389/frsc.2021.710243 · Cited by 107 · see citing works
Vahedi, B., Karimzadeh, M., & Zoraghein, H. (2021) · Nature Communications · doi:10.1038/s41467-021-26742-6 · Cited by 46 · see citing works
Spatiotemporal contact density explains the disparity of COVID-19 spread in urban neighborhoods
Verma, R., Yabe, T., & Ukkusuri, S. V. (2021) · Scientific Reports · doi:10.1038/s41598-021-90483-1 · Cited by 45 · see citing works
A fairness assessment of mobility-based COVID-19 case prediction models
Erfani, A., & Frías-Martínez, V. (2023) · PLoS ONE · doi:10.1371/journal.pone.0292090 · Cited by 22 · see citing works
Assessing Bias in Smartphone Mobility Estimates in Low Income Countries
Milusheva, S., Björkegren, D., & Viotti, L. (2021) · ACM SIGCAS Conference on Computing and Sustainable Societies (COMPASS) · doi:10.1145/3460112.3471968 · Cited by 23 · see citing works
The exciting potential and daunting challenge of using GPS human-mobility data for epidemic modeling
Barreras, F., & Watts, D. J. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00637-0 · Cited by 22 · see citing works
Neighborhood Isolation during the COVID-19 Pandemic
Marlow, T., Makovi, K., & Abrahão, B. (2021) · Sociological Science · doi:10.15195/v8.a9 · Cited by 16 · see citing works
Cheng, S., Zhang, Y., & Shaw, J. (2025) · RSF The Russell Sage Foundation Journal of the Social Sciences · doi:10.7758/rsf.2025.11.1.07 · Cited by 12 · see citing works
Shifting temporal dynamics of human mobility in the United States
Sparks, K. A., Moehl, J., Weber, E. M., Brelsford, C. M., & Rose, A. (2022) · Journal of Transport Geography · doi:10.1016/j.jtrangeo.2022.103295 · Cited by 11 · see citing works
Horn, A. L., Bell, B. M., Bueno, B. G. B., Bahrami, M., Bozkaya, B., Cui, Y., Wilson, J. P., Pentland, A., Moro, E., & de la Haye, K. (2021) · medRxiv · doi:10.1101/2021.10.28.21265634 · Cited by 7 · see citing works
The perils of working with big data, and a SMALL checklist you can use to recognize them
Brave, S. A., Butters, R. A., & Fogarty, M. S. (2021) · Business Horizons · doi:10.1016/j.bushor.2021.06.004 · Cited by 5 · see citing works
Mobility-Based Segregation in U.S. Metropolitan Areas
Zhang, Y., & Cheng, S. (2025) · Demography · doi:10.1215/00703370-12193739 · Cited by 3 · see citing works
Tucker, R., O'Brien, D. T., & Hangen, F. (2024) · CrimRxiv · doi:10.21428/cb6ab371.8efcc1b9 · Cited by 2 · see citing works
Mobile phones and their use to study dynamics of the COVID-19 pandemic
Berke, A., & Larson, K. (2024) · Elsevier eBooks · doi:10.1016/b978-0-323-95646-8.00049-4 · Cited by 1 · see citing works
The perils of working with Big Data and a SMALL framework you can use to avoid them
Brave, S. A., Butters, R. A., & Fogarty, M. S. (2020) · doi:10.21033/wp-2020-35 · Cited by 1 · see citing works
A fairness assessment of mobility-based COVID-19 case prediction modelsPreprint
Erfani, A., & Frías-Martínez, V. (2022) · Research Square · doi:10.21203/rs.3.rs-2164430/v1
Gaps in gender and socioeconomic mobility disparity studies
Gauvin, L. (2024) · Nature Computational Science · doi:10.1038/s43588-024-00667-8
- Authors
- Coston, A., Guha, N., Ouyang, D., Lu, L., Chouldechova, A., & Ho, D. E.
- Affiliations
- Carnegie Mellon University (US)
- Stanford University (US)
- Venue
- Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2021-03
- Pages
- 173-184
- OpenAlex
- W3101729298
- Cited by
- 21
- References
- 47
- Citations by year
- 2025: 2 · 2024: 5 · 2023: 1 · 2022: 3 · 2021: 8 · 2020: 2
- FWCI
- 10.25
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion, cc-by)
- arXiv (Cornell University)
- Funders
- National Science Foundation Graduate Research Fellowship Program (DGE1745016)
- Stanford RISE initiative
- K&L Gates Presidential Fellowship
- Stanford's Institute for Human-Centered Artificial Intelligence
- Topics
- Social Sciences › Transportation › Human Mobility and Location-Based Analysis
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Information Systems › COVID-19 Digital Contact Tracing
- Keywords
- mobility data, bias auditing, administrative data linkage, sampling bias, demographic bias, COVID-19 pandemic response, public health resource allocation, older voters, minority voters, smartphone location data, measurement validity
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
