Predictive Multiplicity in Classification
Open accessThe DOI matches its OpenAlex record; 20 of the 20 references it cites were found in a registry or an index.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Statistical Modeling: The Two Cultures (with comments and a rejoinder by the author)
Breiman, L. (2001) · Statistical Science · doi:10.1214/ss/1009213726 · Cited by 4,340
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Fisher, A., Rudin, C., & Dominici, F. (2018) · arXiv · doi:10.48550/arxiv.1801.01489 · Cited by 1,165
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Lum, K., & Isaac, W. (2016) · Significance · doi:10.1111/j.1740-9713.2016.00960.x · Cited by 611
Explanation, imagination, and confidence in judgment.
Koehler, D. J. (1991) · Psychological Bulletin · doi:10.1037//0033-2909.110.3.499 · Cited by 561
Actionable Recourse in Linear Classification
Ustun, B., Spangher, A., & Liu, Y. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287566 · Cited by 501
Self-citationAequitas: A Bias and Fairness Audit Toolkit
Saleiro, P., Kuester, B., Hinkson, L., London, J., Stevens, A., Anisfeld, A., Rodolfa, K. T., & Ghani, R. (2018) · arXiv · doi:10.48550/arxiv.1811.05577 · Cited by 496
Problem Formulation and Fairness
Passi, S., & Barocas, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287567 · Cited by 283 · cited 1×
“est error) from a collection of models that satisfy a certain constraints (e.g., linear classifiers with equal FPR between groups). Assuming that stakeholders agree on the formulation of this problem [38, 44] and that we can solve it, there is a key issue with this approach: there may be more than one “best" mode…”
BackgroundExplaining Data-Driven Document Classifications1
Martens, D., & Provost, F. (2014) · MIS Quarterly · doi:10.25300/misq/2014/38.1.04 · Cited by 264 · cited 1×
“Credit Scoring : Say we must explain the prediction of a credit scoring model to a person [by producing e.g., a counterfactual explanation 31, 50].”
BackgroundClassification with Fairness Constraints: A Meta-Algorithm with Provable Guarantees
Celis, L. E., Huang, L., Keswani, V., & Vishnoi, N. K. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287586 · Cited by 231 · cited 1×
“, AUC), predict multiple outcomes, or satisfy additional constraints (e.g., group fairness constraints as in Zafar et al., 2019; Celis et al., 2019; Cotter et al., 2019).”
BackgroundProblems with Shapley-value-based explanations as feature importance measures
Kumar, I. E., Venkatasubramanian, S., Scheidegger, C., & Friedler, S. (2020) · arXiv · doi:10.48550/arxiv.2002.11097 · Cited by 228
The Dangers of Post-hoc Interpretability: Unjustified Counterfactual Explanations
Laugel, T., Lesot, M.-J., Marsala, C., Renard, X., & Detyniecki, M. (2019) · Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence · doi:10.24963/ijcai.2019/388 · Cited by 194
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
Designing Contestability: Interaction Design, Machine Learning, and Mental Health
Hirsch, T., Merced, K., Narayanan, S., Imel, Z. E., & Atkins, D. C. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064703 · Cited by 110 · cited 2×
“…and severity of conflicting predictions allows them to challenge key aspects of modern machine learning [as per the principles of contestability 20, 26] – from model selection (e.g., how should we choose between competing models, if we should choose one at all, and who should have a say in…”
BackgroundInterventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment
Barabas, C., Dinakar, K., Ito, J., Virza, M., & Zittrain, J. (2017) · arXiv · doi:10.48550/arxiv.1712.08238 · Cited by 87
Fairwashing: the risk of rationalization
Aïvodji, U., Arai, H., Fortineau, O., Gambs, S., Hara, S., & Tapp, A. (2019) · arXiv · doi:10.48550/arxiv.1901.09749 · Cited by 54
Cited by
Interpretable machine learning: Fundamental principles and 10 grand challenges
Rudin, C., Chen, C., Chen, Z., Huang, H., Semenova, L., & Zhong, C. (2022) · Statistics Surveys · doi:10.1214/21-ss133 · Cited by 958
Underspecification Presents Challenges for Credibility in Modern Machine Learning
D'Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, C., Deaton, J., Eisenstein, J., Hoffman, M. D., et al. (2020) · arXiv · doi:10.48550/arxiv.2011.03395 · Cited by 398
“Our treatment of underspecification is more closely related to work on “Rashomon sets” (Fisher et al., 2019), “predictive multiplicity” (Marx et al., 2019), and methods that search for validationequivalent predictors that are “right for the right reasons” (Ross et al., 2017).”
BackgroundHuman Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks
Lu, Z., & Yin, M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445562 · Cited by 131
On generating trustworthy counterfactual explanations
Del Ser, J., Barredo-Arrieta, A., Díaz-Rodríguez, N., Herrera, F., Saranti, A., & Holzinger, A. (2024) · Information Sciences · doi:10.1016/j.ins.2023.119898 · Cited by 122
On the Existence of Simpler Machine Learning Models
Semenova, L., Rudin, C., & Parr, R. (2022) · 2022 ACM Conference on Fairness Accountability and Transparency · doi:10.1145/3531146.3533232 · Cited by 90
“[31] Charles Marx, Flavio Calmon, and Berk Ustun.”
BackgroundResult
Metadata
- Authors
- Marx, C. T., Calmon, F. P., & Ustun, B.
- Venue
- arXiv (Cornell University)
- Publisher
- Cornell University
- Published
- 2019-09-14
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 1909.06677
- OpenAlex
- W2972763980
- Cited by
- 28
- References
- 20
- Type
- posted content
- Language
- en
- Open access
- green
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.