50 Years of Test (Un)fairness: Lessons for Machine Learning
The DOI matches its Crossref record; 71 of the 71 references it cites were found in a registry or an index.
Abstract
Quantitative definitions of what is unfair and what is fair have been introduced in multiple disciplines for well over 50 years, including in education, hiring, and machine learning. We trace how the notion of fairness has been defined within the testing communities of education and hiring over the past half century, exploring the cultural and social context in which different fairness definitions have emerged. In some cases, earlier definitions of fairness are similar or identical to definitions of fairness in current machine learning research, and foreshadow current formal work. In other cases, insights into what fairness means and how to measure it have largely gone overlooked. We compare past and current notions of fairness along several dimensions, including the fairness criteria, the focus of the criteria (e.g., a test, a model, or its use), the relationship of fairness to individuals, groups, and subgroups, and the mathematical method for measuring fairness (e.g., classification, regression). This work points the way towards future research and measurement of (un)fairness that builds from our modern understanding of fairness while incorporating insights from the past.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“These toolkits have been criticised for offering a 'reductionist understanding of fairness as mathematical conditions' [71, p. 1], and reflect a longer history of attempts to reduce (un)fairness to a metric [72].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [72]
In the citing paper
“Speciically, there is a rich set of fairness-related work in a variety of disciplines, often with concepts that are similar or equal to current ML fairness research [147].”
From Fairness in machine learning: A survey (2024) · cites this paper as [147]
Citations
Cites
Advances in Neural Information Processing Systems 19
(2007) · The MIT Press eBooks · doi:10.52202/085713 · Cited by 15,539
Proceedings of the 25th international conference on Machine learning - ICML '08
(2008) · doi:10.1145/1390156 · Cited by 6,680
(2018) · Cited by 3,510
Semi-supervised learning using Gaussian fields and harmonic functions
Zhu, X., Zoubin Ghahramani, A., & Lafferty, J. (2003) · Cited by 3,509
Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012) · Proceedings of the 3rd Innovations in Theoretical Computer Science Conference · doi:10.1145/2090236.2090255 · Cited by 3,501 · cited 1×
“…A ⊥ R ; it is equivalent when A and R have a bivariate Gaussian • Guion’s definition “people with equal probabilities of success is a special case of Dwork’s [23] individual fairness with the relationship between Y and R (since an ability being measured affects the for the optimal score R ∗”
BackgroundDoyle, T. (2017) · The Information Society · doi:10.1080/01972243.2017.1354593 · Cited by 3,071
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 5×
“bility of success on the job” is a construct that can be meaningfully reasoned about. The fairness literature in both the fields of ML and in testing have also been motivated by causal considerations [32, 41]. Darlington [19] motivate his definition (3) on the basis of a causal relationship between Y and R (since an a…”
BackgroundResultInfluentialFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 3×
“uality of opportunity [32]. That same year, Linn [43] introduced (but did not advocate for) equal Positive Predictive Value (PPV) as a fairness criterion, recognizable as modern day predictive parity [9].4 Under Cleary and Darlington’s conceptions, bias or (un)fairness is a property of the test itself. This is contrar…”
BackgroundResultMitigating Unwanted Biases with Adversarial Learning
Zhang, B. H., Lemoine, B., & Mitchell, M. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278779 · Cited by 1,387 · cited 1×
“or bias in model features. Instead, one approach adopted in ML fairness has been the use of adversarial methods to mitigate the effects of features with undesirable correlations with subgroups, e.g., [6, 69]. 3.4 Model vs. Model Use Section 2 described how the test literature had competing notions of whether fairness…”
MethodInfluentialSelf-citationAlgorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Kusner, M. J., Loftus, J. R., Russell, C., & Silva, R. (2017) · arXiv · doi:10.48550/arxiv.1703.06856 · Cited by 995
Fairness in Criminal Justice Risk Assessments: The State of the Art
Berk, R., Heidari, H., Jabbari, S., Kearns, M., & Roth, A. (2021) · Sociological Methods & Research · doi:10.1177/0049124118782533 · Cited by 988 · cited 1×
“• Peterson and Novick’s “conditional probability and its con-verse” sufficiency [4], equalized odds [32], or conditional procedure accuracy [5], sometimes expressed as the conditional independence A ⊥ D | Y .”
BackgroundInherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
TEST BIAS: PREDICTION OF GRADES OF NEGRO AND WHITE STUDENTS IN INTEGRATED COLLEGES
Cleary, T. A. (1968) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1968.tb00613.x · Cited by 658 · cited 1×
“first time, cast in terms of a formal model for predicting educational outcomes from test scores [10, 11]:”
BackgroundPARTIAL CORRELATION AND CONDITIONAL CORRELATION AS MEASURES OF CONDITIONAL INDEPENDENCE
Baba, K., Shibata, R., & Sibuya, M. (2004) · Australian & New Zealand Journal of Statistics · doi:10.1111/j.1467-842x.2004.00360.x · Cited by 520 · cited 1×
“uivalent to sufficiency in the special case where A, R and Y have a multivariate Gaussian distribution. This is because for this special case the partial correlation ρAY.X = 0 is equivalent to A ⊥Y|R [3]. In general though, we cannot assume even a one way implication, since A ⊥Y|R does not imply ρAY.X = 0 (see [63] fo…”
BackgroundPreventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
Kearns, M., Neel, S., Roth, A., & Wu, Z. S. (2017) · arXiv · doi:10.48550/arxiv.1711.05144 · Cited by 469 · cited 1×
“, skin tone), as recent work has begun to address [27, 39].”
BackgroundDIF DETECTION AND DESCRIPTION: MANTEL‐HAENSZEL AND STANDARDIZATION1,2
Dorans, N. J., & Holland, P. W. (1992) · ETS Research Report Series · doi:10.1002/j.2333-8504.1992.tb01440.x · Cited by 356
Inherent Tradeoffs in the Fair Determination of Risk Scores
(2023) · The Societal Impacts of Algorithmic Decision-Making · doi:10.1145/3603195.3603197 · Cited by 310
Oneto, L., & Chiappa, S. (2020) · Studies in Computational Intelligence · doi:10.1007/978-3-030-43883-8_7 · Cited by 291 · cited 2×
“• Peterson and Novick’s “conditional probability and its con-verse” sufficiency [4], equalized odds [32], or conditional procedure accuracy [5], sometimes expressed as the conditional independence A ⊥ D | Y .”
BackgroundData Decisions and Theoretical Implications when Adversarially Learning Fair Representations
Beutel, A., Chen, J., Zhao, Z., & Chi, E. H. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00075 · Cited by 274
(1989) · Choice Reviews Online · doi:10.5860/choice.27-1601 · Cited by 270
THORNDIKE, R. L. (1971) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1971.tb00907.x · Cited by 265 · cited 1×
“in framing from unfairness to fairness . base rates (i.e., A is not independent of Y ) [24, 62]. test fairness . the contextual use of a test: requiring that the ratio ( TP + FP )/( TP + FN ) be equal for each for test cutoff x ∗ fair if threshold x ∗ 1 is used for group π 1 and threshold x ∗ 2 for…”
BackgroundFriedler, S. A., Scheidegger, C., & Venkatasubramanian, S. (2021) · Communications of the ACM · doi:10.1145/3433949 · Cited by 262
An Intersectional Definition of Fairness
Foulds, J. R., Islam, R., Keya, K. N., & Pan, S. (2020) · 2020 IEEE 36th International Conference on Data Engineering (ICDE) · doi:10.1109/icde48307.2020.00203 · Cited by 233 · cited 1×
“ality can be modeled. This might include, for example, how to quantify fairness along one dimension (e.g., age) conditioned on another dimension (e.g., skin tone), as recent work has begun to address [27, 39]. 6 CONCLUSIONS The spike in interest in test fairness in the 1960s arose during a time of social and political…”
BackgroundAN EVALUATION OF SOME MODELS FOR CULTURE‐FAIR SELECTION
PETERSEN, N. S., & NOVICK, M. R. (1976) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1976.tb00178.x · Cited by 230 · cited 1×
“modeled using a parameter λ : where λ values of -1, 0, and 1 imply Darlington’s definitions (1), (2) Petersen and Novick [52] showed that ( 1 , 1 ) -Thorndikian fairness relaxations of ( λ , λ ) -Thorndikian fairness in which only one of the”
MethodHamidi, F., Scheuerman, M. K., & Branham, S. M. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173582 · Cited by 222
Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy
Verma, S. (2019) · Vikalpa: The Journal for Decision Makers · doi:10.1177/0256090919853933 · Cited by 219
Linn, R. L. (1973) · Review of Educational Research · doi:10.2307/1169933 · Cited by 201 · cited 2×
“In other words: What causes the disparities? (2) Drawing from earlier insights of Guion [30], Thorndike [62], Cole [14], Linn [43], Jones [37], and Peterson & Novick [52] to expand fairness criteria to include model context and use.”
BackgroundANOTHER LOOK AT “CULTURAL FAIRNESS”1
DARLINGTON, R. B. (1971) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1971.tb00908.x · Cited by 199 · cited 5×
“thread of work highlights that different fairness criteria encode different value systems [34], and that quantitative techniques alone cannot answer the question of which to use. In 1971, Darlington [19] urges that the concept of “cultural fairness” be replaced by “cultural optimality”, which takes into account a poli…”
BackgroundInfluentialA method of assessing bias in test items
SCHEUNEMAN, J. (1979) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1979.tb00095.x · Cited by 188 · cited 1×
“Ten years later, the related idea of Differential Item Functioning (DIF) was introduced by Scheuneman in 1979 [58]: “an item is considered unbiased if, for persons with the same ability in the area being measured, the probability of a correct response on the item is the same regardless of the population group membersh…”
BackgroundThe Hardness of Conditional Independence Testing and the Generalised Covariance Measure
Shah, R. D., & Peters, J. (2019) · Apollo (University of Cambridge) · doi:10.17863/cam.45267 · Cited by 176
COLE, N. S. (1973) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1973.tb00802.x · Cited by 168 · cited 3×
“This idea hints at the notion of equal opportunity for those with a given value of Y , an idea which soon was picked up by Darlington [19] and Cole [14].”
BackgroundCleary, T. A., & Hilton, T. L. (1968) · Educational and Psychological Measurement · doi:10.1177/001316446802800106 · Cited by 162 · cited 2×
“In earlier work on test fairness, there was a focus on defining measurements in terms of unfair discrimination and unfair bias, which brought with it the problem of uncovering sources of bias [12].”
BackgroundMethodRunaway Feedback Loops in Predictive Policing
Ensign, D., Friedler, S. A., Neville, S., Scheidegger, C., & Venkatasubramanian, S. (2017) · arXiv · doi:10.48550/arxiv.1706.09847 · Cited by 161 · cited 1×
“The rise of interest in fairness today has corresponded with public interest in the use of machine learning in criminal sentencing and predictive policing, including discussions around compas [16, 20, 42] and PredPol [25, 49].”
BackgroundCritical analysis of the statistical and ethical implications of various definitions of test bias.
Hunter, J. E., & Schmidt, F. L. (1976) · Psychological Bulletin · doi:10.1037/0033-2909.83.6.1053 · Cited by 158 · cited 6×
“(3) Building from earlier insights of 1970s researchers [19, 34, 44] to incorporate quantitative factors for the balance between fairness goals and other goals, such as a value system or a system of ethics.”
BackgroundMethodInfluentialSimple, robust, scalable semi-supervised learning via expectation regularization
Mann, G. S., & McCallum, A. (2007) · International Conference on Machine Learning · doi:10.1145/1273496.1273571 · Cited by 156
The problem of infra-marginality in outcome tests for discrimination
Simoiu, C., Corbett-Davies, S., & Goel, S. (2017) · The Annals of Applied Statistics · doi:10.1214/17-aoas1058 · Cited by 154 · cited 1×
“t we know of. There are conceptual similarities between Jones’ criteria and the constrained ranking problem described by [8], and also between Einhorn’s criterion and concerns about infra-marginality [60]. For a binary classifier, Thorndike’s 1971 group parity criterion is equivalent to requiring that the ratio of pos…”
BackgroundPsychological Testing of American Minorities: Issues and Consequences.
Jorgensen, C. C., & Samuda, R. J. (1977) · Contemporary Sociology · doi:10.2307/2065830 · Cited by 129 · cited 1×
“ties” [66]. The NAACP followed up on this in 1974 by adopting a resolution that demanded “a moratorium on standardized testing wherever such tests have not been corrected for cultural bias” (cited by [56]). Meanwhile, advocates of testing worried that alternatives to testing such as interviews would introduce more sub…”
BackgroundInfluentialWillingham, W. W., & Cole, N. S. (2013) · Routledge eBooks · doi:10.4324/9781315045115 · Cited by 119
Satisfying Real-world Goals with Dataset Constraints
Goh, G., Cotter, A., Gupta, M. R., & Friedlander, M. P. (2016) · arXiv (Cornell University) · doi:10.48550/arxiv.1606.07558 · Cited by 118 · cited 1×
“It is closely related to coverage constraints [29], class mass normalization [70] and expectation regularization [45].”
BackgroundFrom Parity to Preference-based Notions of Fairness in Classification
Zafar, M. B., Valera, I., Gomez-Rodriguez, M., Gummadi, K. P., & Weller, A. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1707.00010 · Cited by 101 · cited 1×
“mise, which in some scenarios might relate to the public’s notion of fairness. We propose that the economics literature on fair division might provide some useful ideas, as has also been suggested by [68]. However, we do heed Darlington’s [19] warning that “a compromise may end up satisfying nobody; psychometricians a…”
BackgroundCole, N. S., & Zieky, M. J. (2001) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.2001.tb01132.x · Cited by 87 · cited 3×
“Perhaps ML practitioners should cautiously take heed from Cole and Zieky’s [15] portrayal of developments in their field:”
BackgroundMethodInfluentialFairness in Educational Assessment and Measurement
Dorans, N. J., & Cook, L. L. (2016) · Routledge eBooks · doi:10.4324/9781315774527 · Cited by 83
Vars, F. E., & Bowen, W. G. (1998) · Cited by 76
InclusiveFaceNet: Improving Face Attribute Detection with Race and Gender Diversity
Ryu, H. J., Adam, H., & Mitchell, M. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1712.00193 · Cited by 75
Self-citationMethodological considerations relevant to discrimination in employment testing.
Einhorn, H. J., & Bass, A. R. (1971) · Psychological Bulletin · doi:10.1037/h0030871 · Cited by 74 · cited 1×
“in framing from unfairness to fairness . base rates (i.e., A is not independent of Y ) [24, 62]. test fairness . the contextual use of a test: requiring that the ratio ( TP + FP )/( TP + FN ) be equal for each for test cutoff x ∗ fair if threshold x ∗ 1 is used for group π 1 and threshold x ∗ 2 for…”
BackgroundDichotomization, Partial Correlation, and Conditional Independence
Vargha, A., Rudas, T., Delaney, H. D., & Maxwell, S. E. (1996) · Journal of Educational and Behavioral Statistics · doi:10.3102/10769986021003264 · Cited by 68 · cited 1×
“…π 1 and threshold x ∗ 2 for group π 2 of the test score R given the target variable Y . notion of equal opportunity for those with a given value of Y , an in terms of the correlation ρ AR between the demographic variable holds for every subgroup, then ρ AR = ρ AY / ρ RY [63]. that ρ AR = ρ AY .”
BackgroundEmployment tests and discriminatory hiring
GUION, R. M. (1966) · Industrial Relations A Journal of Economy and Society · doi:10.1111/j.1468-232x.1966.tb00449.x · Cited by 59
Cojuharenco, I., & Patient, D. (2013) · Journal of Occupational and Organizational Psychology · doi:10.1111/joop.12023 · Cited by 52 · cited 1×
“[35]), and a shift in focus from outcomes to inputs and processes [13].”
BackgroundIN SEARCH OF FAIR SELECTION PROCEDURES
Linn, R. L. (1976) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1976.tb00181.x · Cited by 52 · cited 3×
“We agree with Linn’s [44] argument from 1976 that values encoded by technical definitions should be made explicit.”
BackgroundUTILITIES AND THE ISSUE OF FAIRNESS IN A DECISION THEORETIC MODEL FOR SELECTION
SAWYER, R. L., COLE, N. S., & COLE, J. W. L. (1976) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1976.tb00182.x · Cited by 47 · cited 1×
“[57], in a foreshadowing of the compas debate of 2016:”
BackgroundAn Expected Utility Model for "Optimal" Selection
Petersen, N. S. (1976) · Journal of Educational Statistics · doi:10.3102/10769986001004333 · Cited by 38
Cleary, T. A. (1966) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1966.tb00529.x · Cited by 37 · cited 1×
“al distributions of test scores and ground truth scores for subgroups π1 and π2 are shown by the axes. first time, cast in terms of a formal model for predicting educational outcomes from test scores [10, 11]: A test is biasedfor members of a subgroup of the population if, in the prediction of a criterion for which th…”
BackgroundTOWARDS EQUALIZING EDUCATIONAL AND EMPLOYMENT OPPORTUNITY
Novick, M. R., & PETERSEN, N. S. (1976) · Journal of Educational Measurement · doi:10.1111/j.1745-3984.1976.tb00183.x · Cited by 30 · cited 3×
“the 1970s indicate an increasing concern with how fairness criteria obscure “the fundamental problem, which is to find some rational basis for providing compensatory treatment for the disadvantaged” [48].”
Backgroundvon Davier, M., & Bennett, R. E. (2017) · Methodology of educational measurement and assessment · doi:10.1007/978-3-319-58689-2 · Cited by 29
Psychological Tests: Uses and Abuses
Anastasi, A. (1961) · Teachers College Record The Voice of Scholarship in Education · doi:10.1177/016146816106200502 · Cited by 21
The War Against Testing: A Current Status Report
Williams, R. L., Dotson, W., Don, P., & Williams, W. S. (1980) · The Journal of Negro Education · doi:10.2307/2295085 · Cited by 18 · cited 1×
“Responding to these concerns, the Association of Black Psychologists formed in 1969 immediately published “A Petition of Concerns”, calling for a moratorium on standardized tests “(which are used) to maintain and justify the practice of systematically denying economic opportunities” [66].”
BackgroundAn Expected Utility Model for "Optimal" Selection
Petersen, N. S. (1976) · Journal of Educational Statistics · doi:10.2307/1164987 · Cited by 16
Ranking with Fairness Constraints
Celis, L. E., Straszak, D., & Vishnoi, N. K. (2017) · arXiv (Cornell University) · doi:10.4230/lipics.icalp.2018.28 · Cited by 16
The implications of the Civil Rights Act of 1964 for psychological assessment in industry.
Ash, P. (1966) · American Psychologist · doi:10.1037/h0023906 · Cited by 13 · cited 2×
“found that Myart actually passed the test, and so declined to rule on the fairness of the test [2].”
BackgroundDichotomization, Partial Correlation, and Conditional Independence
Vargha, A., Rudas, T., Delaney, H. D., Maxwell, S. E., Vargha, A., & Rudas, T. (1996) · Journal of Educational and Behavioral Statistics · doi:10.2307/1165272 · Cited by 12
Philosophical Perspectives on Fairness in Educational Assessment
Zwick, R., & Dorans, N. J. (2016) · Routledge eBooks · doi:10.4324/9781315774527-17 · Cited by 10
Moderated Regression and Equal Opportunity
Jones, M. B. (1973) · Educational and Psychological Measurement · doi:10.1177/001316447303300306 · Cited by 6 · cited 2×
“which includes sufficiency and separation [4]. for Darlington’s criterion (2) and Jones’ criteria “at position n ” and 3: Relationships between testing criteria and ML’s independence criteria Thorndike’s criterion is reminiscent of fair division in economics.”
BackgroundContributions to the Quantitative Assessment of Item, Test, and Score Fairness
Dorans, N. J. (2017) · Methodology of educational measurement and assessment · doi:10.1007/978-3-319-58689-2_7 · Cited by 6
Legal Aspects of Test Fairness
Phillips, S. (2016) · Routledge eBooks · doi:10.4324/9781315774527-16 · Cited by 5 · cited 1×
“More examples of court rulings on fairness are given by [53, 71].”
BackgroundThe Overprediction Phenomenon among Black Collegians: Some Prelinimary Considerations.
Thomas, C. L. (1973) · Cited by 3
Employment Tests and Discriminatory Hiring
Guion, R. M. (1967) · Journal of Occupational and Environmental Medicine · doi:10.1097/00043764-196705000-00037 · Cited by 3 · cited 6×
“• Similarly, their “equal probability and its converse” A ⊥ R ; it is equivalent when A and R have a bivariate Gaussian • Guion’s definition “people with equal probabilities of success is a special case of Dwork’s [23] individual fairness with the relationship between Y and R (since an ability being measured affects t…”
BackgroundInfluentialAN EXPLORATORY STUDY OF PROGRAMMED TESTS
Cleary, T. A., Linn, R. L., & Rock, D. A. (1966) · ETS Research Bulletin Series · doi:10.1002/j.2333-8504.1966.tb00541.x · Cited by 3
Fairness Feuds: Competing Conceptions of Title VII Discriminatory TestingPreprint
West-Faulcon, K. (2011) · SSRN Electronic Journal · Cited by 2
Research works printed but not linked: 11
- Beutel Alex, 2017, Chi
- Buolamwini Joy, 2018, Conference on Fairness, Accountability and Transparency. 77--91
- James, 2018, Foulds and Shimei Pan
- Friedler Sorelle A, 2016, On the (im) possibility of fairness. arXiv preprint arXiv: 1609.07236 · arXiv
- Hoffmann Anna Lauren, Spaces for the Future
- Jencks Christopher, 1998, Racial bias in testing. The Black-White test score gap 55
- 10.3102/00346543043002139 · doi:10.3102/00346543043002139 · Paper page
- National Council on Measurement in Education NCME (Ed.). 1976. Journal of Education Measurement. 13 1 (1976). National Council on Measurement in Education NCME (Ed.). 1976. Journal of Education Measurement. 13 1 (1976).
- Phillips S E, Fairness in Educational Assessment and Measurement
- Rice Mitchell F, 1994, Race Norming, Validity Generalization, and Employment Testing, Handbook of Public Personnel Administration, 58, 451
- Zwick Rebecca, Fairness in Educational Assessment and Measurement
Cited by
A Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2022) · ACM Computing Surveys · doi:10.1145/3457607 · Cited by 5,812
“rvey on Bias and Fairness in Machine Learning 11 4.2 Definitions of Fairness In [16], authors studied fairness definitions in political philosophy and tried to tie them to machinelearning. Authors in [60] studied the 50-year history of fairness definitions in the areas of education and machine-learning. In [122], auth…”
BackgroundInfluentialA Survey on Bias and Fairness in Machine Learning
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019) · arXiv · doi:10.48550/arxiv.1908.09635 · Cited by 5,812
Mitigating bias in algorithmic hiring: evaluating claims and practices
Raghavan, M., Barocas, S., Kleinberg, J., & Levy, K. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372828 · Cited by 845
“This has led to a decades-long line of literature seeking to measure and mitigate the observed disparities (see [50] for a survey).”
BackgroundBaker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
“In historical perspective, Hutchinson and Mitchell (2019) point out that many formalized definitions of fairness have precedents from the 1960s and 1970s as researchers worked to minimize bias in standardized testing.”
BackgroundAlgorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Metadata
- Authors
- Hutchinson, B., & Mitchell, M.
- ORCID
- Mitchell, Margaret
- Venue
- Proceedings of the Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2019-01-29
- Pages
- 49-58
- OpenAlex
- W4288617757
- Cited by
- 297
- References
- 68
- Citations by year
- 2026: 23 · 2025: 45 · 2024: 38 · 2023: 61 · 2022: 49 · 2021: 45 · 2020: 25 · 2019: 8
- FWCI
- 33.19
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Social Sciences › Law › Discrimination and Equality Law
- Social Sciences › Decision Sciences › Management Science and Operations Research › Psychometric Methodologies and Testing
- Keywords
- test fairness, educational testing, algorithmic fairness, fairness criteria, fairness measurement, subgroup fairness, history of testing
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.