A Nutritional Label for Rankings
Open accessThe DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.
Abstract
Algorithmic decisions often result in scoring and ranking individuals to determine credit worthiness, qualifications for college admissions and employment, and compatibility as dating partners. While automatic and seemingly objective, ranking algorithms can discriminate against individuals and protected groups, and exhibit low diversity. Furthermore, ranked results are often unstable -- small changes in the input data or in the ranking methodology may lead to drastic changes in the output, making the result uninformative and easy to manipulate. Similar concerns apply in cases where items other than individuals are ranked, including colleges, academic departments, or products. Despite the ubiquity of rankers, there is, to the best of our knowledge, no technical work that focuses on making rankers transparent.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“In each case, attending to end-use provides a richer design space and new opportunities for improvements and safeguards.”
From Toward situated interventions for algorithmic equity: lessons from the field (2020) · cites this paper as [63]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
"Why Should I Trust You?": Explaining the Predictions of Any Classifier
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016) · Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2939672.2939778 · Cited by 17,555
“Why Should I Trust You?”: Explaining the Predictions of Any Classifier
Ribeiro, M., Singh, S., & Guestrin, C. (2016) · Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations · doi:10.18653/v1/n16-3020 · Cited by 6,682
Foucault, M. (2005) · doi:10.4324/9780203996645 · Cited by 3,281
FA*IR: A Fair Top-k Ranking Algorithm
Zehlike, M., Bonchi, F., Castillo, C., Hajian, S., Megahed, M., & Baeza-Yates, R. (2017) · Proceedings of the 2017 ACM on Conference on Information and Knowledge Management · doi:10.1145/3132847.3132938 · Cited by 496 · cited 4×
“view of the Fairness widget in our example presents the output of three fairness measures: FA*IR [14], proportion [15], and our own pairwise measure.”
BackgroundMethodInfluentialMeasuring Fairness in Ranked Outputs
Yang, K., & Stoyanovich, J. (2017) · Proceedings of the 29th International Conference on Scientific and Statistical Database Management · doi:10.1145/3085504.3085526 · Cited by 378 · cited 3×
“The Fairness widget quanti es whether the ranked output exhibits statistical parity (one interpretation of fairness) with respect to one or more sensitive attributes, such as gender or race of individuals [13].”
BackgroundMethodSelf-citationMeasuring discrimination in algorithmic decision making
Žliobaitė, I. (2017) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-017-0506-1 · Cited by 272 · cited 2×
“view of the Fairness widget in our example presents the output of three fairness measures: FA*IR [14], proportion [15], and our own pairwise measure.”
BackgroundDiversity in Big Data: A Review
Drosou, M., Jagadish, H. V., Pitoura, E., & Stoyanovich, J. (2017) · Big Data · doi:10.1089/big.2016.0054 · Cited by 140 · cited 4×
“The Diversity widget shows diversity with respect to a set of demographic categories of individuals, or a set of categorical attributes of other kinds of items [2].”
BackgroundMethodInfluentialSelf-citationBLACK BOX TINKERING: Beyond Disclosure in Algorithmic EnforcementPreprint
Perel, M., & Elkin-Koren, N. (2016) · SSRN Electronic Journal · doi:10.2139/ssrn.2741513 · Cited by 134
Proceedings of the 14th International Conference on Extending Database Technology
Ailamaki, A., Amer-Yahia, S., Pate, J., Risch, T., Senellart, P. P., & Stoyanovich, J. (2011) · doi:10.1145/1951365 · Cited by 110
Self-citationOnline Set Selection with Fairness and Diversity Constraints
Stoyanovich, J., Yang, K., & Jagadish, H. V. (2018) · Movebank · doi:10.5441/002/edbt.2018.22 · Cited by 86 · cited 1×
“Each widget addresses an essential aspect of transparency and interpretability, and is based on our recent technical work on fairness and diversity [2, 9, 11, 13], transparency [10], and stability (ongoing) in algorithmic rankers.”
BackgroundSelf-citation
Research works printed but not linked: 4
- [1] Danielle K. Citron and Frank A. Pasquale. 2014. The Scored Society: Due Process for Automated Predictions. Washington Law Review 89 (2014). -
- [4] Joshua A. Kroll, Joanna Huey, Solon Barocas, Edward W. Felten, Joel R. Reidenberg, David G. Robinson, and Harlan Yu. 2017. Accountable Algorithms. University of Pennsylvania Law Review 165 (2017). -
- [8] Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cedric Langbort. 2014. Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms. Data and Discrimination: Converting Critical Concerns into Productive Inquiry (2014). -
- [10] Julia Stoyanovich and Ellen P. Goodman. August 5, 2016. Revealing Algorithmic Rankers. Freedom to Tinker (August 5, 2016). http://freedom-to-tinker.com/2016/ 08/05/revealing-algorithmic-rankers/ -
Cited by
Fairness and Abstraction in Sociotechnical Systems
Selbst, A. D., Boyd, D., Friedler, S. A., Venkatasubramanian, S., & Vertesi, J. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287598 · Cited by 1,631
“In fact, a number of recent papers have proposed ideas along these lines [12, 27, 37, 55, 58, 60, 71].”
BackgroundBender, E. M., & Friedman, B. (2018) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00041 · Cited by 854
“…“datasheets for datasets,” looking at AI more broadly (but including NLP); Chmielinski and colleagues at the MIT Media Lab propose “dataset nutrition labels”; 19 and Yang et al. (2018) describe “Ranking Facts,” a series of widgets that allow a user to explore how attributes influence a ranking.”
BackgroundAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
“Similarly, documenting model performance, both overall and within subgroups, is crucial to effective evaluation and can also help check decision systems against some of the fairness definitions from Section 4 (Mitchell et al. 2018, Stoyanovich 2018, Yang et al. 2018).”
BackgroundExplainability fact sheets: a framework for systematic assessment of explainable approaches
Sokol, K., & Flach, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372870 · Cited by 264
“• quickly grasp properties of a given method to choose an appropriate approach for a desired use case given the unified design of the Fact Sheets (as with food nutrition labels [56] the users know what to expect and how to navigate them).”
MethodResultThe Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
““Ranking Facts” is a web-based system that generates a “nutrition label” for scoring and ranking algorithms based on factors or “widgets” to communicate an algorithm’s methodology or output [24] .”
Background
Metadata
- Authors
- Yang, K., Stoyanovich, J., Asudeh, A., Howe, B., Jagadish, H., & Miklau, G.
- Affiliations
- Drexel University (US)
- University of Michigan (US)
- University of Washington (US)
- University of Massachusetts Amherst (US)
- Venue
- Proceedings of the 2018 International Conference on Management of Data
- Publisher
- ACM
- Published
- 2018-05-27
- Pages
- 1773-1776
- ISSN
- 0730-8078 (ISSN-L)
- OpenAlex
- W2798497570
- Cited by
- 103
- References
- 14
- Citations by year
- 2026: 5 · 2025: 8 · 2024: 3 · 2023: 17 · 2022: 14 · 2021: 21 · 2020: 14 · 2019: 16 · 2018: 4
- FWCI
- 24.62
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM SIGMOD International Conference on Management of Data (SIGMOD) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- Figshare (submittedVersion, cc-by)
- arXiv (Cornell University)
- University of Illinois Chicago
- Funders
- Microsoft (Cascadia Urban Analytics Cooperative)
- National Science Foundation (1741047,1740996,1741022,1741254)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Social Sciences › Economics, Econometrics and Finance › Economics and Econometrics › Game Theory and Voting Systems
- Keywords
- ranking algorithms, algorithmic transparency, algorithmic discrimination, protected groups, ranking stability, credit scoring, university admission, employment decisions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: A Nutritional Label for Rankings. https://app.cochat.ai/papers/10.1145/3183713.3193568
- The paper
- Yang, K., Stoyanovich, J., Asudeh, A., Howe, B., Jagadish, H., & Miklau, G. (2018). A Nutritional Label for Rankings. Proceedings of the 2018 International Conference on Management of Data. https://doi.org/10.1145/3183713.3193568
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.