CoChat

Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity

Gardner, Joshua, Yu, Renzhe, Nguyen, Quan, Brooks, Christopher, Kizilcec, Rene · 2023 · 2023 ACM Conference on Fairness Accountability and Transparency
Cited by 22

The DOI matches its Crossref record; 14 of the 14 references it cites were found in a registry or an index.

Abstract

Modern machine learning increasingly supports paradigms that are multi-institutional (using data from multiple institutions during training) or cross-institutional (using models from multiple institutions for inference), but the empirical effects of these paradigms are not well understood. This study investigates cross-institutional learning via an empirical case study in higher education. We propose a framework and metrics for assessing the utility and fairness of student dropout prediction models that are transferred across institutions. We examine the feasibility of cross-institutional transfer under real-world data- and model-sharing constraints, quantifying model biases for intersectional student identities, characterizing potential disparate impact due to these biases, and investigating the impact of various cross-institutional ensembling approaches on fairness and overall model performance. We perform this analysis on data representing over 200,000 enrolled students annually from four universities without sharing training data between institutions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3593013.3594107

Cites

14 of 14 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 14 references

Research works printed but not linked: 61

  1. 10.1145/2976749.2978318 · DOI · Paper page
  2. Orevaoghene Ahia, Julia Kreutzer, and Sara Hooker. 2021. The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation. arXiv preprint arXiv:2110.03036 (2021). · arXiv
  3. Izzat Alsmadi, ZW Taylor, and Joshua Childs. 2020. US News & World Report Best Colleges rankings: Which institutional metrics contribute to sustained stratification?Scientometrics 124, 3 (2020), 1851–1869.
  4. 10.1145/2983323.2983351 · DOI · Paper page
  5. Noah Arthurs and AJ Alvero. 2020. Whose Truth is the “Ground Truth”? College Admissions Essays and Bias in Word Vector Evaluation Methods. In Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020). 342–349.
  6. Lovenoor Aulck, Dev Nambi, Nishant Velagapudi, Joshua Blumenstock, and Jevin West. 2019. Mining University Registrar Records to Predict First-Year Undergraduate Attrition. In Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019). 9–18.
  7. 10.35542/osf.io/pbmvz · DOI
  8. 10.1007/s11162-019-09546-y · DOI · Paper page
  9. 10.1177/23328584211037630 · DOI · Paper page
  10. 10.1007/978-3-319-19773-9_6 · DOI
  11. Andrew P Bradley. 1997. The use of the area under the ROC curve in the evaluation of machine learning algorithms. Pattern recognition 30, 7 (1997), 1145–1159.
  12. Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accuracy disparities in commercial gender classification. In Conference on fairness, accountability and transparency. PMLR, 77–91.
  13. 10.1145/3170358.3170410 · DOI · Paper page
  14. 10.1145/3442188.3445879 · DOI
  15. 10.1145/3306618.3314236 · DOI · Paper page
  16. Andrew Cotter, Maya Gupta, Heinrich Jiang, Nathan Srebro, Karthik Sridharan, Serena Wang, Blake Woodworth, and Seungil You. 2019. Training well-generalizing classifiers for fairness metrics and other data-dependent constraints. In International Conference on Machine Learning. PMLR, 1397–1405.
  17. Gerben W. Dekker, Mykola Pechenizkiy, and Jan M. Vleeshouwers. 2009. Predicting students drop out: A case study. In Proceedings of the 2nd International Conference on Educational Data Mining (EDM 2009). 41–50.
  18. 10.1007/978-3-030-52237-7_11 · DOI · Paper page
  19. 10.1145/3303772.3303794 · DOI
  20. 10.1145/3303772.3303838 · DOI · Paper page
  21. Zilah EISESTEIN. 1979. Combahee River collective: A Black Feminist Statement. Capitalist Patriarchy and the Case for Socialist Feminism (1979).
  22. James Fogarty, Ryan S Baker, and Scott E Hudson. 2005. Case studies in the use of ROC curve analysis for sensor-based estimates in human computer interaction. In Proceedings of Graphics Interface 2005. 129–136.
  23. 10.1145/3303772.3303791 · DOI · Paper page
  24. Joshua P Gardner Zoran Popovi and Ludwig Schmidt. [n. d.]. Subgroup Robustness Grows On Trees: An Empirical Baseline Investigation. In Advances in Neural Information Processing Systems.
  25. James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (ROC) curve.Radiology 143, 1 (1982), 29–36.
  26. Tatsunori B Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. 2018. Fairness without demographics in repeated loss minimization. arXiv preprint arXiv:1806.08010 (2018). · arXiv
  27. Nicholas W Hillman, David A Tandberg, and Jacob PK Gross. 2014. Performance funding in higher education: Do financial incentives impact college completions?The journal of higher education 85, 6 (2014), 826–857.
  28. Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. 2020. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058 (2020). · arXiv
  29. Qian Hu and Huzefa Rangwala. 2020. Towards Fair Educational Data Mining: A Case Study on Detecting At-risk Students. In Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020). 431–437.
  30. Stephen Hutt, Margo Gardner, Angela L. Duckworth, and Sidney K. D’Mello. 2019. Evaluating Fairness and Generalizability in Models Predicting On-Time Graduation from College Applications. In Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019).
  31. 10.18608/jla.2014.11.3 · DOI · Paper page
  32. Erik Jones, Shiori Sagawa, Pang Wei Koh, Ananya Kumar, and Percy Liang. 2020. Selective Classification Can Magnify Disparities Across Groups. arXiv preprint arXiv:2010.14134 (2020). · arXiv
  33. Peter Kairouz, H Brendan McMahan, Brendan Avent, Aurélien Bellet, Mehdi Bennis, Arjun Nitin Bhagoji, Kallista Bonawitz, Zachary Charles, Graham Cormode, Rachel Cummings, 2019. Advances and open problems in federated learning. arXiv preprint arXiv:1912.04977 (2019). · arXiv
  34. 10.1109/ICDMW.2011.83 · DOI · Paper page
  35. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems 30 (2017).
  36. Fereshte Khani and Percy Liang. 2020. Feature Noise Induces Loss Discrepancy Across Groups. In International Conference on Machine Learning. PMLR, 5209–5219.
  37. René F. Kizilcec and Hansol Lee. 2022. Algorithmic fairness in education. In The Ethics of Artificial Intelligence in Education. Routledge, 174–202.
  38. 10.1145/3328526.3329621 · DOI · Paper page
  39. 10.1002/aehe.3205 · DOI · Paper page
  40. 10.1145/3386527.3406755 · DOI
  41. 10.1145/3152714 · DOI
  42. Hansol Lee and René F. Kizilcec. 2020. Evaluation of Fairness Trade-offs in Predicting Student Success. arxiv:2007.00088 [cs.CY] · arXiv
  43. Xingyu Li Difan Song Miaozhe Han Yu Zhang and René F Kizilcec. 2021. On the limits of algorithmic prediction across the globe. (2021). https://arxiv.org/abs/2103.15212 · arXiv
  44. 10.18653/v1/W19-4401 · DOI · Paper page
  45. Yuetian Luo and Zachary A. Pardos. 2018. Diagnosing University Student Subject Proficiency and Predicting Degree Completion in Vector Space. In Proceedings of the Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans, LA, USA.
  46. 10.1086/669570 · DOI · Paper page
  47. Julian Matschinske, Julian Späth, Reza Nasirigerdeh, Reihaneh Torkzadehmahani, Anne Hartebrodt, Balázs Orbán, Sándor Fejér, Olga Zolotareva, Mohammad Bakhtiari, Béla Bihari, 2021. The FeatureCloud AI Store for Federated Learning in Biomedicine and Beyond. arXiv preprint arXiv:2105.05734 (2021). · arXiv
  48. Giulio Morina, Viktoriia Oliinyk, Julian Waton, Ines Marusic, and Konstantinos Georgatzis. 2019. Auditing and achieving intersectional fairness in classification problems. arXiv preprint arXiv:1911.01468 (2019). · arXiv
  49. 10.1111/bjet.12156 · DOI · Paper page
  50. Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. 2018. Scalable private learning with pate. arXiv preprint arXiv:1802.08908 (2018). · arXiv
  51. Dana Pessach, Tamir Tassa, and Erez Shmueli. 2021. Fairness-Driven Private Collaborative Machine Learning. arXiv preprint arXiv:2109.14376 (2021). · arXiv
  52. Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 (2019). · arXiv
  53. Mark Schneider and Lu Yin. 2011. The High Cost of Low Graduation Rates: How Much Does Dropping Out of College Really Cost?Technical Report. · doi:10.1037/e537282012-001
  54. 10.1145/3442188.3445865 · DOI · Paper page
  55. Kai Ming Ting and Ian H Witten. 1997. Stacked Generalization: when does it work? (1997).
  56. 10.3102/00346543045001089 · doi:10.3102/00346543045001089 · Paper page
  57. S. Truth. 1851. Ain’t I A Woman?Speech delivered at Women’s Rights Convention, Akron, Ohio.
  58. David H Wolpert. 1992. Stacked generalization. Neural networks 5, 2 (1992), 241–259.
  59. Forest Yang, Moustapha Cisse, and Sanmi Koyejo. 2020. Fairness with Overlapping Groups. arXiv preprint arXiv:2006.13485 (2020). · arXiv
  60. 10.1145/3430895.3460139 · DOI · Paper page
  61. Renzhe Yu, Qiujie Li, Christian Fischer, Shayan Doroudi, and Di Xu. 2020. Towards Accurate and Fair Prediction of College Success: Evaluating Different Sources of Student Data. In Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020). 292–301.

Cited by

See all 22

Metadata

Authors
Gardner, J., Yu, R., Nguyen, Q., Brooks, C., & Kizilcec, R.
Affiliations
  • University of Washington, USA
  • Teachers College, Columbia University, USA
  • University of British Columbia, Canada
  • University of Michigan, USA
  • Cornell University, USA
Venue
2023 ACM Conference on Fairness Accountability and Transparency
Publisher
ACM
Published
2023-06-12
Pages
1664-1684
Cited by
22
References
14

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity. https://app.cochat.ai/papers/10.1145/3593013.3594107
The paper
Gardner, J., Yu, R., Nguyen, Q., Brooks, C., & Kizilcec, R. (2023). Cross-Institutional Transfer Learning for Educational Models: Implications for Model Performance, Fairness, and Equity. 2023 ACM Conference on Fairness Accountability and Transparency. https://doi.org/10.1145/3593013.3594107

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.