Measuring Fairness in Ranked Outputs
Open accessAbstract
Ranking and scoring are ubiquitous. We consider the setting in which an institution, called a ranker, evaluates a set of individuals based on demographic, behavioral or other characteristics. The final output is a ranking that represents the relative quality of the individuals. While automatic and therefore seemingly objective, rankers can, and often do, discriminate against individuals and systematically disadvantage members of protected groups. This warrants a careful study of the fairness of a ranking scheme, to enable data science for social good applications, among others.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Considerations of fairness have been actively studied in the context of rankings and recommender systems.
“Considerations of fairness have been actively studied in the context of rankings and recommender systems.”
From Fairness in machine learning: A survey (2024) · cites this paper as [302]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
UCI Machine Learning Repository
Asuncion, A. (2007) · Medical Entomology and Zoology · Cited by 24,190 · see citing works
Cumulated gain-based evaluation of IR techniques
Järvelin, K., & Kekäläinen, J. (2002) · ACM Transactions on Information Systems · doi:10.1145/582415.582418 · Cited by 4,841 · see citing works
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · see citing works
(2013) · Cited by 1,126 · see citing works
On the (im)possibility of fairness
Friedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2016) · arXiv · doi:10.48550/arxiv.1609.07236 · Cited by 247 · see citing works
A survey on measuring indirect discrimination in machine learningPreprint
Žliobaitė, I. (2015) · arXiv.org · doi:10.48550/arxiv.1511.00148 · Cited by 157 · see citing works
Discovering Unwarranted Associations in Data-Driven Applications with the FairTest Testing ToolkitPreprint
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J., Humbert, M., Juels, A., & Huang, L. (2015) · arXiv (Cornell University) · Cited by 19 · see citing works
Making interval-based clustering rank-aware
Stoyanovich, J., Amer-Yahia, S., & Milo, T. (2011) · International Conference on Extending Database Technology (EDBT) · doi:10.1145/1951365.1951417 · Cited by 12 · see citing works
Self-citation
Flagged first, then by citations
Research works printed but not linked: 2
- Danielle K. Citron and Frank A. Pasquale. 2014. The Scored Society: Due Process for Automated Predictions. Washington Law Review 89 (2014). http://papers.ssrn. com/sol3/papers.cfm?abstract_id=2376209
- Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Rei- denberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165 (2017). http://papers.ssrn.com/sol3/ papers.cfm?abstract_id=2765268
Cited by
Sorted by citations
Celik, I. (2023) · Computers in Human Behavior · doi:10.1016/j.chb.2022.107468 · Cited by 1,012 · see citing works
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845 · see citing works
Role of fairness, accountability, and transparency in algorithmic affordance
Shin, D., & Park, Y. J. (2019) · Computers in Human Behavior · doi:10.1016/j.chb.2019.04.019 · Cited by 643 · see citing works
Fairness in machine learning: A survey
Caton, S., & Haas, C. (2024) · ACM Computing Surveys · doi:10.1145/3616865 · Cited by 646 · see citing works
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · see citing works
Equity of Attention: Amortizing Individual Fairness in Rankings
Biega, A. J., Gummadi, K. P., & Weikum, G. (2018) · The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval · doi:10.1145/3209978.3210063 · Cited by 337 · see citing works
Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting
De-Arteaga, M., Romanov, A., Wallach, H., Chayes, J., Borgs, C., Chouldechova, A., Geyik, S., Kenthapadi, K., & Kalai, A. T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287572 · Cited by 337 · see citing works
Classification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · see citing works
Investigating the Impact of Gender on Rank in Resume Search Engines
Chen, L., Ma, R., Hannák, A., & Wilson, C. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174225 · Cited by 155 · see citing works
Fairness in Machine Learning: A Survey
Caton, S., & Haas, C. (2020) · arXiv · doi:10.48550/arxiv.2010.04053 · Cited by 646 · see citing works
A Nutritional Label for Rankings
Yang, K., Stoyanovich, J., Asudeh, A., Howe, B., Jagadish, H., & Miklau, G. (2018) · Proceedings of the 2018 International Conference on Management of Data · doi:10.1145/3183713.3193568 · Cited by 103 · see citing works
Self-citationQuantifying the Impact of User Attentionon Fair Group Representation in Ranked Lists
Sapiezynski, P., Zeng, W., E Robertson, R., Mislove, A., & Wilson, C. (2019) · Companion Proceedings of The 2019 World Wide Web Conference · doi:10.1145/3308560.3317595 · Cited by 92 · see citing works
Diversity and Inclusion Metrics in Subset Selection
Mitchell, M., Baker, D., Moorosi, N., Denton, R., Hutchinson, B., Hanna, A., Gebru, T., & Morgenstern, J. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375832 · Cited by 87 · see citing works
Anti-discrimination learning: a causal modeling-based framework
Zhang, L., & Wu, X. (2017) · International Journal of Data Science and Analytics · doi:10.1007/s41060-017-0058-x · Cited by 48 · see citing works
<scp>FAIR</scp> : Fairness‐aware information retrieval evaluation
Gao, R., Ge, Y., & Shah, C. (2022) · Journal of the Association for Information Science and Technology · doi:10.1002/asi.24648 · Cited by 19 · see citing works
Algorithmic monoculture and social welfare
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603202
- Authors
- Yang, K., & Stoyanovich, J.
- Affiliations
- Drexel University (US)
- Venue
- Proceedings of the 29th International Conference on Scientific and Statistical Database Management
- Publisher
- ACM
- Published
- 2017-06-27
- Pages
- 1-6
- ISSN
- 1551-6393 (ISSN-L)
- OpenAlex
- W2544318541
- Cited by
- 378
- References
- 8
- Citations by year
- 2026: 18 · 2025: 27 · 2024: 51 · 2023: 58 · 2022: 45 · 2021: 76 · 2020: 41 · 2019: 40 · 2018: 15 · 2017: 6 · 2012: 1
- FWCI
- 69.63
- Type
- conference paper
- Language
- en
- Open access
- bronze
- doi.org
- Locations
- International Conference on Scientific and Statistical Database Management (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1464327, 1539856)
- US-Israel Binational Science Foundation (2014391)
- Topics
- Social Sciences › Safety Research › Ethics and Social Impacts of AI
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Social Sciences › Law › Discrimination and Equality Law
- Keywords
- algorithmic fairness, ranking, discrimination, protected groups, fairness metrics, ranking scheme, demographic bias
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
