The DOI matches its Crossref record; 9 of the 9 references it cites were found in a registry or an index.
Evaluating the Fairness of Predictive Student Models Through Slicing Analysis
Abstract
Predictive modeling has been a core area of learning analytics research over the past decade, with such models currently deployed in a variety of educational contexts from MOOCs to K-12. However, analyses of the differential effectiveness of these models across demographic, identity, or other groups has been scarce. In this paper, we present a method for evaluating unfairness in predictive student models. We define this in terms of differential accuracy between subgroups, and measure it using a new metric we term the Absolute Between-ROC Area (ABROCA). We demonstrate the proposed method through a gender-based "slicing analysis" using five different models replicated from other works and a dataset of 44 unique MOOCs and over four million learners. Our results demonstrate (1) significant differences in model fairness according to (a) statistical algorithm and (b) feature set used; (2) that the gender imbalance ratio, curricular area, and specific course used for a model all display significant association with the value of the ABROCA statistic; and (3) that there is not evidence of a strict tradeoff between performance and fairness. This work provides a framework for quantifying and understanding how predictive models might inadvertently privilege, or disparately impact, different student subgroups. Furthermore, our results suggest that learning analytics researchers and practitioners can use slicing analysis to improve model fairness without necessarily sacrificing performance.1
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Hanley, J. A., & McNeil, B. J. (1982) · Radiology · doi:10.1148/radiology.143.1.7063747 · Cited by 21,856 · cited 3×
“…future work detailing the asymptotic properties of this statistic, which is a function of the ROC curve, which itself has well-established statistical properties [18], would aid in developing procedures for inference (for example, providing estimates of standard errors and significance for ABROCA).”
BackgroundMethodBarocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536
Certifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
AI can be sexist and racist — it’s time to make it fair
Zou, J., & Schiebinger, L. (2018) · Nature · doi:10.1038/d41586-018-05707-8 · Cited by 765
Facts Are More Important Than Novelty: Replication in the Education Sciences
Makel, M. C., & Plucker, J. A. (2014) · Educational Researcher · doi:10.3102/0013189x14545513 · Cited by 609 · cited 1×
“First, there has been only limited replication research in the field of learning analytics to date, with the replication rate (the percentage of studies which are replications) estimated at approximately 0.13% in the field of education [25].”
BackgroundInfluentialA comparative study of fairness-enhancing interventions in machine learning
Friedler, S. A., Scheidegger, C., Venkatasubramanian, S., Choudhary, S., Hamilton, E. P., & Roth, D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287589 · Cited by 601
There is a blind spot in AI research
Crawford, K., & Calo, R. (2016) · Nature · doi:10.1038/538311a · Cited by 580
A multidisciplinary survey on discrimination analysis
Romei, A., & Ruggieri, S. (2014) · The Knowledge Engineering Review · doi:10.1017/s0269888913000039 · Cited by 288
Cited by
Explainable Artificial Intelligence in education
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y.-S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022) · Computers and Education: Artificial Intelligence · doi:10.1016/j.caeai.2022.100074 · Cited by 863
Baker, R. S., & Hawn, A. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00285-9 · Cited by 993
Self-citationChounta, I.-A., Bardone, E., Raudsep, A., & Pedaste, M. (2022) · International Journal of Artificial Intelligence in Education · doi:10.1007/s40593-021-00243-5 · Cited by 441
Personalized Education in the Artificial Intelligence Era: What to Expect Next
Maghsudi, S., Lan, A., Xu, J., & van der Schaar, M. (2021) · IEEE Signal Processing Magazine · doi:10.1109/msp.2021.3055032 · Cited by 341
Learning analytics dashboards: the past, the present and the future
Verbert, K., Ochoa, X., De Croon, R., Dourado, R. A., & De Laet, T. (2020) · Proceedings of the Tenth International Conference on Learning Analytics & Knowledge · doi:10.1145/3375462.3375504 · Cited by 129
Metadata
- Authors
- Gardner, J., Brooks, C., & Baker, R.
- Venue
- Proceedings of the 9th International Conference on Learning Analytics & Knowledge
- Publisher
- ACM
- Published
- 2019-03-04
- Pages
- 225-234
- OpenAlex
- W2915273119
- Cited by
- 185
- References
- 9
- FWCI
- 21.48
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Evaluating the Fairness of Predictive Student Models Through Slicing Analysis. https://app.cochat.ai/papers/10.1145/3303772.3303791
- The paper
- Gardner, J., Brooks, C., & Baker, R. (2019). Evaluating the Fairness of Predictive Student Models Through Slicing Analysis. Proceedings of the 9th International Conference on Learning Analytics & Knowledge. https://doi.org/10.1145/3303772.3303791
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
