CoChat
Sign Up
PreprintCited by 687Ethics and Social Impacts of AIExplainable Artificial Intelligence (XAI)Discrimination and Equality Law

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions

Open access
Mitchell, Shira, Potash, Eric, Barocas, Solon, D'Amour, Alexander, Lum, Kristian·2018·arXiv
DOI 10.48550/arxiv.1811.07867arXiv 1811.07867
Open this paper in CoChat — free account

Abstract

A recent flurry of research activity has attempted to quantitatively define "fairness" for decisions based on statistical and machine learning (ML) predictions. The rapid growth of this new field has led to wildly inconsistent terminology and notation, presenting a serious challenge for cataloguing and comparing definitions. This paper attempts to bring much-needed order. First, we explicate the various choices and assumptions made---often implicitly---to justify the use of prediction-based decisions. Next, we show how such choices and assumptions can raise concerns about fairness and we present a notationally consistent catalogue of fairness definitions from the ML literature. In doing so, we offer a concise reference for thinking through the choices, assumptions, and fairness considerations of prediction-based decision systems.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.1811.07867

Cites

3 of 3 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Parity on different metrics across disaggregated population subgroups corresponds to how fairness is often defined.

    “Parity on the different metrics across disaggregated population subgroups corresponds to how fairness is often defined [37, 48].”

    From Model Cards for Model Reporting (2019) · cites this paper as [37]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 96

  1. [1][1] Michelle Alexander. The New Jim Crow . New Press, 2012. -
  2. [2][2] Rico Angell, Brittany Johnson, Yuriy Brun, and Alexandra Meliou. Themis: Automatically testing software for discrimination. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , pages 871-875. ACM, 2018. -
  3. [4][4] Chelsea Barabas, Karthik Dinakar, Joichi Ito, Madars Virza, and Jonathan Zittrain. Interventions over predictions: Reframing the ethical debate for actuarial risk assessment. In Conference on Fairness, Accountability and Transparency (FAT*) , 2018. -
  4. [7][7] James O. Berger. Statistical Decision Theory and Bayesian Analysis . Springer Series in Statistics. Springer New York, 1985. -
  5. [8][8] Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: the state of the art. 2017. https://arxiv.org/abs/1703.09207 . - · arXiv
  6. [10][10] Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on Fairness, Accountability and Transparency (FAT*) , pages 7791, 2018. -
  7. [11][11] Toon Calders and Sicco Verwer. Three naive bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery , 21(2):277-292, 2010. -
  8. [12][12] Alexandra Chouldechova. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data , 5(2):153-163, 2017. https://www.andrew.cmu.edu/user/achoulde/ files/disparate\_impact.pdf . -
  9. [13][13] Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Conference on Fairness, Accountability and Transparency (FAT*) , pages 134-148, 2018. -
  10. [14][14] Alexandra Chouldechova and Max G'Sell. Fairer and more accurate, but for whom? 2017. https: //arxiv.org/abs/1707.00046 . -
  11. [15][15] Alexandra Chouldechova and Aaron Roth. The frontiers of fairness in machine learning. 2018. https: //arxiv.org/abs/1810.08810 . -
  12. [16][16] Anne T. Cleary. Test bias: Validity of the scholastic aptitude test for negro and white students in integrated colleges. ETS Research Bulletin Series , 1966(2):i-23, 1966. -
  13. [18][18] Sam Corbett-Davies and Sharad Goel. The measure and mismeasure of fairness: A critical review of fair machine learning, 2018. https://arxiv.org/abs/1808.00023 . - · arXiv
  14. [19][19] Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 797-806. ACM, 2017. -
  15. [20][20] Amanda Coston, Alan Mishler, Edward H Kennedy, and Alexandra Chouldechova. Counterfactual risk assessments, evaluation, and fairness. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , pages 582-593, 2020. -
  16. [21][21] Marsha Courchane, David Nebhut, and David Nickerson. Lessons learned: Statistical techniques and fair lending. Journal of Housing Research , pages 277-295, 2000. -
  17. [23][23] David Danks and Alex John London. Algorithmic bias in autonomous systems. In IJCAI , pages 4691-4697, 2017. -
  18. [24][24] Richard B. Darlington. Another look at 'cultural fairness'. Journal of Educational Measurement , 8(2):71-82, 1971. -
  19. [25][25] Jared N Day. Credit, capital and community: Informal banking in immigrant communities in the united states, 1880-1924. Financial History Review , 9(1):65-78, 2002. -
  20. [26][26] Maria De-Arteaga, Artur Dubrawski, and Alexandra Chouldechova. Learning under selective labels in the presence of expert consistency. 2018. Presented at the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2018), Stockholm, Sweden. -
  21. [29][29] Roel Dobbe, Sarah Dean, Thomas Gilbert, and Nitin Kohli. A broader view on bias in automated decision-making: Reflecting on epistemology and dynamics. 2018. Presented at the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2018), Stockholm, Sweden. -
  22. [31][31] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. In Proceedings of the 3rd innovations in theoretical computer science conference , pages 214-226. ACM, 2012. -
  23. [33][33] Hillel J. Einhorn and Alan R. Bass. Methodological considerations relevant to discrimination in employment testing. Psychological Bulletin , 75(4):261, 1971. -
  24. [34][34] Virginia Eubanks. Automating Inequality: How High-Tech Tools Profile, Police, and Punish the Poor . St. Martin's Press, 2018. -
  25. [36][36] Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining , pages 259-268, 2015. -
  26. [39][39] Sorelle A. Friedler, Carlos Scheidegger, and Suresh Venkatasubramanian. On the (im)possibility of fairness. 2016. https://arxiv.org/abs/1609.07236 . - · arXiv
  27. [40][40] Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth. A comparative study of fairness-enhancing interventions in machine learning. 2018. https://arxiv.org/abs/1802.04422 . - · arXiv
  28. [42][42] Andreas Fuster, Paul Goldsmith-Pinkham, Tarun Ramadorai, and Ansgar Walther. Predictably unequal? the effects of machine learning on credit markets. 2018. https://papers.ssrn.com/sol3/ papers.cfm?abstract\_id=3072038 . -
  29. [43][43] Sainyam Galhotra, Yuriy Brun, and Alexandra Meliou. Fairness testing: testing software for discrimination. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , pages 498-510. ACM, 2017. -
  30. [44][44] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume´ e III, and Kate Crawford. Datasheets for datasets. 2018. https://arxiv.org/abs/1803.09010 . - · arXiv
  31. [46][46] Andrew Gelman, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. Bayesian Data Analysis, Third Edition . Chapman & Hall/CRC Texts in Statistical Science. Taylor & Francis, 2013. -
  32. [48][48] Ben Green. 'fair' risk assessments: A precarious approach for criminal justice reform. 2018. Presented at the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2018), Stockholm, Sweden. -
  33. [49][49] Ben Green and Lily Hu. The myth in the methodology: Towards a recontextualization of fairness in machine learning. 2018. Presented at the Machine Learning: The Debates workshop at the 35th International Conference on Machine Learning (ICML). -
  34. [50][50] James D. Greiner and Donald B. Rubin. Causal effects of perceived immutable characteristics. Review of Economics and Statistics , 93(3):775-785, 2011. -
  35. [51][51] Bernard E. Harcourt. Against prediction: Profiling, policing, and punishing in an actuarial age . University of Chicago Press, 2008. -
  36. [52][52] Moritz Hardt, Eric Price, Nati Srebro, et al. Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3315-3323, 2016. -
  37. [54][54] Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. The dataset nutrition label: A framework to drive higher data quality standards. 2018. https://arxiv.org/abs/ 1805.03677 . -
  38. [55][55] Lily Hu and Yiling Chen. A short-term intervention for long-term fairness in the labor market. 2018. https://arxiv.org/abs/1712.00064 . - · arXiv
  39. [56][56] Lily Hu and Yiling Chen. Welfare and distributional impacts of fair classification. 2018. Presented at the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2018), Stockholm, Sweden. -
  40. [57][57] John E Hunter and Frank L Schmidt. Critical analysis of the statistical and ethical implications of various definitions of test bias. Psychological Bulletin , 83(6):1053, 1976. -
  41. [58][58] Ben Hutchinson and Margaret Mitchell. 50 years of test (un)fairness: Lessons for machine learning. In Conference on Fairness, Accountability and Transparency (FAT*) , 2019. -
  42. [59][59] Guido W. Imbens and Donald B. Rubin. Causal Inference in Statistics, Social, and Biomedical Sciences . Causal Inference for Statistics, Social, and Biomedical Sciences: An Introduction. Cambridge University Press, 2015. -
  43. [60][60] John W Jackson and Tyler J VanderWeele. Decomposition analysis to identify intervention targets for reducing disparities. Epidemiology , 29(6):825-835, 2018. -
  44. [61][61] James E. Johndrow and Kristian Lum. An algorithm for removing sensitive information: application to race-independent recidivism prediction. 2017. https://arxiv.org/abs/1703.04957 . - · arXiv
  45. [63][63] Nathan Kallus and Angela Zhou. Residual unfairness in fair machine learning from prejudiced data. In Proceedings of the 35th International Conference on Machine Learning (ICML) , 2018. -
  46. [64][64] Faisal Kamiran and Toon Calders. Classifying without discriminating. In Computer, Control and Communication, 2009. IC4 2009. 2nd International Conference on , pages 1-6. IEEE, 2009. -
  47. [65][65] Faisal Kamiran and Toon Calders. Classifying without discriminating. In 2009 2nd International Conference on Computer, Control and Communication , pages 1-6. IEEE, 2009. -
  48. [66][66] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. Discrimination aware decision tree learning. In Data Mining (ICDM), 2010 IEEE 10th International Conference on , pages 869-874. IEEE, 2010. -
  49. [67][67] Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. Fairness-aware learning through regularization approach. In Data Mining Workshops (ICDMW), 2011 IEEE 11th International Conference on , pages 643-650. IEEE, 2011. -
  50. [68][68] Samuel Karlin and Herman Rubin. The theory of decision procedures for distributions with monotone likelihood ratio. The Annals of Mathematical Statistics , pages 272-299, 1956. -
  51. [70][70] Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Sch¨ olkopf. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 656-666, 2017. -
  52. [71][71] Michael P. Kim, Omer Reingold, and Guy N. Rothblum. Fairness through computationally-bounded awareness. In Advances in Neural Information Processing Systems (NeurIPS) , 2018. -
  53. [72][72] Jon Kleinberg, Himabindu Lakkaraju, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. Human decisions and machine predictions. The quarterly journal of economics , 133(1):237-293, 2017. -
  54. [73][73] Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. Algorithmic fairness. In AEA Papers and Proceedings , volume 108, pages 22-27, 2018. -
  55. [74][74] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. 2016. https://arxiv.org/abs/1609.05807 . - · arXiv
  56. [75][75] John Logan Koepke and David G Robinson. Danger ahead: Risk assessment and the future of bail reform. Washington Law Review , 2018. -
  57. [77][77] Robert Koulish. Immigration detention in the risk classification assessment era. Connecticut Public Interest Law Journal , 16(1), November 2016. https://ssrn.com/abstract=2865972 . -
  58. [78][78] Matt J. Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. In Advances in Neural Information Processing Systems (NeurIPS) , pages 4066-4076, 2017. -
  59. [79][79] Himabindu Lakkaraju, Jon Kleinberg, Jure Leskovec, Jens Ludwig, and Sendhil Mullainathan. The selective labels problem: Evaluating algorithmic predictions in the presence of unobservables. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 275-284. ACM, 2017. -
  60. [82][82] Zachary Lipton, Julian McAuley, and Alexandra Chouldechova. Does mitigating ml's impact disparity require treatment disparity? In Advances in Neural Information Processing Systems (NeurIPS) , pages 8136-8146, 2018. -
  61. [83][83] Roderick J Little. To model or not to model? competing modes of inference for finite population sampling. Journal of the American Statistical Association , 99(466):546-556, 2004. -
  62. [84][84] Lydia T. Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. Delayed impact of fair machine learning. In Proceedings of the 35th International Conference on Machine Learning (ICML) , 2018. -
  63. [85][85] Kristian Lum and William Isaac. To predict and serve? Significance , 13(5):14-19, 2016. -
  64. [87][87] Caleb H. Miles, Maya Petersen, and Mark J. van der Laan. Causal inference for a single group of causally-connected units under stratified interference. 2017. https://arxiv.org/abs/1710.09588 . - · arXiv
  65. [90][90] Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. 2018. https://arxiv.org/abs/1810.03993 . - · arXiv
  66. [91][91] Razieh Nabi and Ilya Shpitser. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 2018, page 1931. NIH Public Access, 2018. -
  67. [94][94] Rodrigo Ochigame, Chelsea Barabas, Karthik Dinakar, Madars Virza, and Joichi Ito. Beyond legitimation: Rethinking fairness, interpretability, and accuracy in machine learning. 2018. Presented at the Machine Learning: The Debates workshop at the 35th International Conference on Machine Learning (ICML). -
  68. [96][96] Institutes of Medicine (IOM). Unequal Treatment: Confronting Racial and Ethnic Disparities in Health Care . Washington, DC: National Academies Press, 2003. -
  69. [97][97] Elizabeth L Ogburn, Tyler J VanderWeele, et al. Causal diagrams for interference. Statistical science , 29(4):559-578, 2014. -
  70. [99][99] Cathy O'Neil and Hanna Gunn. Near term artificial intelligence and the ethical matrix, 2018. Book chapter, to appear. -
  71. [100][100] Samir Passi and Solon Barocas. Problem formulation and fairness. In Conference on Fairness, Accountability and Transparency (FAT*) , 2019. -
  72. [101][101] Judea Pearl. Causality . Cambridge University Press, 2009. -
  73. [102][102] Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. Discrimination-aware data mining. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 560-568. ACM, 2008. -
  74. [103][103] Emilija Perkovi´ c, Johannes Textor, Markus Kalisch, and Marloes H Maathuis. A complete generalized adjustment criterion. 2015. https://arxiv.org/abs/1507.01524 . - · arXiv
  75. [104][104] Nancy S. Petersen and Melvin R. Novick. An evaluation of some models for culture-fair selection. Journal of Educational Measurement , 13(1):3-29, 1976. -
  76. [105][105] Eric Potash, Joe Brew, Alexander Loewi, Subhabrata Majumdar, Andrew Reece, Joe Walsh, Eric Rozier, Emile Jorgenson, Raed Mansour, and Rayid Ghani. Predictive modeling for public health: Preventing childhood lead poisoning. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 2039-2047. ACM, 2015. -
  77. [106][106] Stephan Rabanser, Stephan G¨ unnemann, and Zachary C. Lipton. Failing loudly: An empirical study of methods for detecting dataset shift. 2018. https://arxiv.org/abs/1810.11953 . - · arXiv
  78. [108][108] John Rawls. A Theory of Justice . Harvard University Press, 1971. -
  79. [109][109] Guy N. Rothblum and Gal Yona. Probably approximately metric-fair learning. In Proceedings of the 35th International Conference on Machine Learning (ICML) , 2018. Appearing in the 5th Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2018), Stockholm, Sweden. -
  80. [110][110] Donald B Rubin. Inference and missing data. Biometrika , 63(3):581-592, 1976. -
  81. [111][111] Donald B. Rubin. Causal inference using potential outcomes: Design, modeling, decisions. Journal of the American Statistical Association , 100(469):322-331, 2005. -
  82. [112][112] Cynthia Rudin. Please stop explaining black box models for high stakes decisions. ArXiv , abs/1811.10154, 2018. - · arXiv
  83. [113][113] Andrew D. Selbst, danah boyd, Sorelle Friedler, Suresh Venkatasubramanian, and Janet Vertesi. Fairness and abstraction in sociotechnical systems. In Conference on Fairness, Accountability and Transparency (FAT*) , 2019. -
  84. [115][115] Camelia Simoiu, Sam Corbett-Davies, and Sharad Goel. The problem of infra-marginality in outcome tests for discrimination. The Annals of Applied Statistics , 11(3):1193-1216, 2017. -
  85. [118][118] Harini Suresh and John V Guttag. A framework for understanding unintended consequences of machine learning. 2019. https://arxiv.org/abs/1901.10002 . - · arXiv
  86. [119][119] Latanya Sweeney. Discrimination in online ad delivery. Queue , 11(3):10, 2013. -
  87. [120][120] Robert L. Thorndike. Concepts of culture-fairness. Journal of Educational Measurement , 8(2):63-70, 1971. -
  88. [122][122] Rhema Vaithianathan, Tim Maloney, Emily Putnam-Hornstein, and Nan Jiang. Children in the public benefit system at risk of maltreatment: Identification via predictive modeling. American journal of preventive medicine , 45(3):354-359, 2013. -
  89. [123][123] Tyler J. VanderWeele and Miguel A. Hern´ an. Results on differential and dependent measurement error of the exposure and the outcome using signed directed acyclic graphs. American journal of epidemiology , 175(12):1303-1310, 2012. -
  90. [124][124] Tyler J. VanderWeele and Whitney R. Robinson. On causal interpretation of race in regressions adjusting for confounding and mediating variables. Epidemiology (Cambridge, Mass.) , 25(4):473, 2014. -
  91. [126][126] Sahil Verma and Julia Rubin. Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare) , pages 1-7. IEEE, 2018. -
  92. [127][127] Larry Wasserman. All of Statistics: A Concise Course in Statistical Inference . Springer Texts in Statistics. Springer New York, 2010. -
  93. [128][128] Ke Yang, Julia Stoyanovich, Abolfazl Asudeh, Bill Howe, HV Jagadish, and Gerome Miklau. A nutritional label for rankings. In Proceedings of the 2018 International Conference on Management of Data , pages 1773-1776. ACM, 2018. -
  94. [130][130] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th International Conference on World Wide Web , pages 1171-1180. International World Wide Web Conferences Steering Committee, 2017. -
  95. [131][131] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. Fairness constraints: Mechanisms for fair classification. AISTATS, 2017. -
  96. [133][133] Junzhe Zhang and Elias Bareinboim. Fairness in decision-making-the causal explanation formula. In 32nd AAAI Conference on Artificial Intelligence , 2018.

Cited by

Sorted by citations

Authors
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K.
Affiliations
Google (United States) (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
Volume
8
Issue
1
Pages
141-163
ISSN
2326-8298 (ISSN-L)
DOI
10.48550/arxiv.1811.07867
arXiv
1811.07867
OpenAlex
W3101206394
Cited by
687
References
3
Citations by year
2026: 104 · 2025: 119 · 2024: 146 · 2023: 136 · 2022: 85 · 2021: 51 · 2020: 25 · 2019: 18 · 2018: 2 · 2012: 1
FWCI
14.33
Type
journal article
Language
en
Open access
Licence
Locations
  • Annual Review of Statistics and Its Application (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Social Sciences › Safety Research › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Social Sciences › Law › Discrimination and Equality Law
Keywords
algorithmic fairness, fairness definitions, automated decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review