Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.
Abstract
Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, an approach to audit such models without probing the black-box model API or pre-defining features to audit. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by the black-box models. We compare the mimic model trained with distillation to a second, un-distilled transparent model trained on ground truth outcomes, and use differences between the two models to gain insight into the black-box model. We demonstrate the approach on four data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 1×
“Model distillation [6, 13] was originally designed to distill knowledge from a large, complex model (the teacher model) to a faster, simpler model (the student model) without significant loss in prediction accuracy.”
BackgroundSelf-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Self-citationCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Self-citationA Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Research works printed but not linked: 17
- Julius Adebayo and Lalana Kagal . 2016 . Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop . Julius Adebayo and Lalana Kagal. 2016. Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop .
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017.
- Susan Athey , Julie Tibshirani , and Stefan Wager . 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 ( 2017 ). Susan Athey, Julie Tibshirani, and Stefan Wager. 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 (2017). · arXiv
- Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS . Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS .
- Alexandra Chouldechova and Max G'Sell . 2017 . Fairer and more accurate, but for whom? . In FATML Workshop . Alexandra Chouldechova and Max G'Sell. 2017. Fairer and more accurate, but for whom?. In FATML Workshop .
- Trevor J Hastie and Robert J Tibshirani. 199 0. Generalized additive models .CRC press. Trevor J Hastie and Robert J Tibshirani. 1990. Generalized additive models .CRC press.
- Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2014 . Distilling the Knowledge in a Neural Network . In NIPS Deep Learning and Representation Learning Workshop . Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop .
- Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML . Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML .
- Michael P Kim , Amirata Ghorbani , and James Zou . 2018 . Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). Michael P Kim, Amirata Ghorbani, and James Zou. 2018. Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). · arXiv
- Manuel Lingo and Gerhard Winkler . 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation ( 2008 ). Manuel Lingo and Gerhard Winkler. 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation (2008).
- Francisco Louzada , Anderson Ara , and Guilherme B Fernandes . 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science ( 2016 ). Francisco Louzada, Anderson Ara, and Guilherme B Fernandes. 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science (2016).
- Lucas Mentch and Giles Hooker . 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research ( 2016 ). Lucas Mentch and Giles Hooker. 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research (2016).
- 10.1016/j.csda.2008.08.007 · DOI
- Sarah Tan , Rich Caruana , Giles Hooker , and Albert Gordo . 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 ( 2018 ). Sarah Tan, Rich Caruana, Giles Hooker, and Albert Gordo. 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 (2018). · arXiv
- Sandra Wachter , Brent Mittelstadt , and Chris Russell . 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology ( 2018 ). Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology (2018).
- Hao Wang , Berk Ustun , and Flavio P. Calmon . 2018 . On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory . Hao Wang, Berk Ustun, and Flavio P. Calmon. 2018. On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory .
- Zhe Zhang and Daniel B. Neill . 2017 . Identifying Significant Predictive Bias in Classifiers. In FATML Workshop . Zhe Zhang and Daniel B. Neill. 2017. Identifying Significant Predictive Bias in Classifiers. In FATML Workshop .
Cited by
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“Tan et al. [103]. expanded on the notion of distilling intricate black-box models into transparent ones known as iGAMs through model distillation.”
BackgroundFooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Metadata
- Authors
- Tan, S., Caruana, R., Hooker, G., & Lou, Y.
- Venue
- Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2018-12-27
- Pages
- 303-310
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4289097366
- Cited by
- 125
- References
- 21
- FWCI
- 3.43
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation. https://app.cochat.ai/papers/10.1145/3278721.3278725
- The paper
- Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018). Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3278721.3278725
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.