Studying up: reorienting the study of algorithmic fairness around issues of power
Open accessThe DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Abstract
Research within the social sciences and humanities has long characterized the work of data science as a sociotechnical process, comprised of a set of logics and techniques that are inseparable from specific social norms, expectations and contexts of development and use. Yet all too often the assumptions and premises underlying data analysis remain unexamined, even in contemporary debates about the fairness of algorithmic systems. This blindspot exists in part because the methodological toolkit used to evaluate the fairness of algorithmic systems remains limited to a narrow set of computational and legal modes of analysis. In this paper, we expand on Elish and Boyd's [17] call for data scientists to develop more robust frameworks for understanding their work as situated practice by examining a specific methodological debate within the field of anthropology, frequently referred to as the practice of "studying up". We reflect on the contributions that the call to "study up" has made in the field of anthropology before making the case that the field of algorithmic fairness would similarly benefit from a reorientation "upward". A case study from our own work illustrates what it looks like to reorient one's research questions "up" in a high-profile debate regarding the fairness of an algorithmic system - namely, pretrial risk assessment in American criminal law. We discuss the limitations of contemporary fairness discourse with regard to pretrial risk assessment before highlighting the insights gained when we reframe our research questions to focus on those who inhabit positions of power and authority within the U.S. court system. Finally, we reflect on the challenges we have encountered in implementing data science projects that "study up". In the process, we surface new insights and questions about what it means to ethically engage in data science work that directly confronts issues of power and authority.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631 · cited 1×
“Researchers in the field of data science have begun to call for more careful consideration of the projects that researchers agree to take on, after thoughtful evaluation of the social context and the politics of their intervention [49].”
BackgroundLepri, B., Oliver, N., Letouzé, E., Pentland, A., & Vinck, P. (2018) · Philosophy & Technology · doi:10.1007/s13347-017-0279-x · Cited by 709 · cited 1×
“These controversies have given rise to an influential community of researchers from both academia and industry who have formed a new regulatory science [31] under the rubric of "fair, accountable, and transparent algorithms" [13, 33].”
BackgroundWhere fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · cited 5×
“As various scholars have argued, such an uncritical acceptance of default assumptions inevitably leads to discriminatory design in algorithmic systems by reproducing ideas which normalize social hierarchies and legitimize violence against marginalized groups [6, 30].”
BackgroundInfluentialSituating methods in the magic of Big Data and AI
Elish, M. C., & boyd, D. (2018) · Communication Monographs · doi:10.1080/03637751.2017.1375130 · Cited by 409
The Intuitive Appeal of Explainable Machines
Selbst, A. D., & Barocas, S. (2018) · doi:10.2139/ssrn.3126971 · Cited by 392
Greene, D., Hoffmann, A. L., & Stark, L. (2019) · Proceedings of the Annual Hawaii International Conference on System Sciences · doi:10.24251/hicss.2019.258 · Cited by 386 · cited 1×
“In light of these concerns, some scholars have suggested that designers of algorithmic systems embrace a process-driven approach to algorithmic justice, one that explicitly recognizes the active and crucial role that the data scientist plays in constructing meaning from data [10, 17, 24, 48].”
BackgroundAlgorithms in practice: Comparing web journalism and criminal justice
Christin, A. (2017) · Big Data & Society · doi:10.1177/2053951717718855 · Cited by 379 · cited 1×
“Mass pretrial incarceration represents a case of "institutional de-coupling" whereby the everyday practices of the courtroom diverge from state and federal laws regarding pretrial release [9].”
BackgroundDesign Justice: towards an intersectional feminist framework for design theory and practice
Costanza-Chock, S. (2018) · Proceedings of DRS · doi:10.21606/drs.2018.679 · Cited by 325
Cited by
Regulating ChatGPT and other Large Generative AI Models
Hacker, P., Engel, A., & Mauer, M. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594067 · Cited by 466
Fairness perceptions of algorithmic decision-making: A systematic review of the empirical literature
Starke, C., Baleis, J., Keller, B., & Marcinkowski, F. (2022) · Big Data & Society · doi:10.1177/20539517221115189 · Cited by 316
“However, addressing the societal implications of (un)fair ADM requires more than mere technological solutions (Barabas et al., 2020; Sloane and Moss, 2019): designing and implementing fair algorithms calls for a thorough empirical understanding of when and why citizens perceive ADM to be (un)fair.”
BackgroundBetween Subjectivity and Imposition: Power Dynamics in Data Annotation for Computer Vision
Miceli, M., Schuessler, M., & Yang, T. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3415186 · Cited by 149
Harms from Increasingly Agentic Algorithmic Systems
Chan, A., Salganik, R., Markelius, A., Pang, C., Rajkumar, N., Krasheninnikov, D., Langosco, L., He, Z., Duan, Y., Carroll, M., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594033 · Cited by 149
Rethinking Fairness: An Interdisciplinary Survey of Critiques of Hegemonic ML Fairness Approaches
Weinberg, L. (2022) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.13196 · Cited by 127
Studying Up Machine Learning Data: Why Talk About Bias When We Mean Power?
Miceli, M., Posada, J., & Yang, T. (2022) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3492853 · Cited by 125
Suresh, H., Gomez, S. R., Nam, K. K., & Satyanarayan, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445088 · Cited by 117
“Interpretability methods that instead shift their gaze upwards and focus on alerting affected stakeholders to potentially discriminatory or arbitrary behaviors by the algorithm might provide much stronger evidence to fight against the reigning power differentials [13].”
BackgroundBringing the People Back In: Contesting Benchmark Machine Learning Datasets
Denton, R., Hanna, A., Amironesei, R., Smart, A., Nicole, H., & Scheuerman, M. K. (2020) · arXiv · doi:10.48550/arxiv.2007.07399 · Cited by 100
“…up” by pointing our social scientific tools towards those with economic, social, and technological power to understand how their norms, values, and practices and power relations shape the data which undergirds everyday sociotechnical systems ( Nader, 1972; Forsythe, 1999; Barabas et al., 2020).”
BackgroundAI’s Regimes of Representation: A Community-centered Study of Text-to-Image Models in South Asia
Qadri, R., Shelby, R., Bennett, C. L., & Denton, R. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594016 · Cited by 85
Organizational Governance of Emerging Technologies: AI Adoption in Healthcare
Kim, J. Y., Boag, W., Gulamali, F., Hasan, A., Hogg, H. D. J., Lifson, M., Mulligan, D., Patel, M., Raji, I. D., Sehgal, A., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594089 · Cited by 55
“practices,” [2] aligning with recent calls for “studying up” into the details of institutional decision-making as a means of holding powerful actors accountable [3].”
BackgroundEveryday Uncertainty: How Blind People Use GenAI Tools for Information Access
Tang, X., Abdolrahmani, A., Gergle, D., & Piper, A. M. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713433 · Cited by 30
Kawakami, A., Coston, A., Heidari, H., Holstein, K., & Zhu, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686989 · Cited by 24
Search atlas: Visualizing divergent search results across geopolitical borders
Ochigame, R., & Ye, K. (2021) · Designing Interactive Systems Conference 2021 · doi:10.1145/3461778.3462032 · Cited by 23
Tang, X., Lima, G., Jiang, L., Simko, L., & Zou, Y. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3710935 · Cited by 17
ACROCPoLis: A Descriptive Framework for Making Sense of Fairness
Aler Tubella, A., Coelho Mollo, D., Dahlgren Lindström, A., Devinney, H., Dignum, V., Ericson, P., Jonsson, A., Kampik, T., Lenaerts, T., Mendez, J. A., et al. (2023) · 2023 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3593013.3594059 · Cited by 8
Prediction and Punishment: Critical Report on Carceral AI
Pruss, D., Pullen-Blasnik, H., Stevens, N., Rahman, S., Belitz, C., Stapleton, L., Dharmaraj, M. G., Aizeki, M., Molnar, P., Pinch, A., et al. (2025) · doi:10.2139/ssrn.5017321 · Cited by 5
Tang, X., Li, J., & Wu, S. (2026) · Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790405 · Cited by 4
“Theseissuesinaccessibilitydatasetannotation alignwithbroader calls in critical AI research to shift attention to the underlying systems of power in AI development [5, 70].”
Diagnosing and Addressing Emergent Harms in the Design Process of Public AI and Algorithmic Systems
Nouws, S., Martinez De Rituerto De Troya, Í., Dobbe, R., & Janssen, M. (2023) · Proceedings of the 24th Annual International Conference on Digital Government Research · doi:10.1145/3598469.3598557 · Cited by 4
The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes
Hall, S. M., Dalal, S., Sefala, R., Yuehgoh, F., Alaagib, A., Hamzaoui, I., Ishida, S., Magomere, J., Crais, L., Salama, A., et al. (2025) · arXiv · doi:10.48550/arxiv.2502.05961
Metadata
- Authors
- Barabas, C., Doyle, C., Rubinovitz, J., & Dinakar, K.
- Venue
- Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2020-01-27
- Pages
- 167-176
- OpenAlex
- W3003599720
- Cited by
- 108
- References
- 13
- FWCI
- 16.97
- Type
- conference paper
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.