Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation
The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index.
Abstract
Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, an approach to audit such models without probing the black-box model API or pre-defining features to audit. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by the black-box models. We compare the mimic model trained with distillation to a second, un-distilled transparent model trained on ground truth outcomes, and use differences between the two models to gain insight into the black-box model. We demonstrate the approach on four data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Towards A Rigorous Science of Interpretable Machine Learning
Doshi-Velez, F., & Kim, B. (2017) · arXiv · doi:10.48550/arxiv.1702.08608 · Cited by 3,121
Fair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159
Buciluǎ, C., Caruana, R., & Niculescu-Mizil, A. (2006) · Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/1150402.1150464 · Cited by 2,068 · cited 1×
“Model distillation [6, 13] was originally designed to distill knowledge from a large, complex model (the teacher model) to a faster, simpler model (the student model) without significant loss in prediction accuracy.”
BackgroundSelf-citationPredicting good probabilities with supervised learning
Niculescu-Mizil, A., & Caruana, R. (2005) · Proceedings of the 22nd international conference on Machine learning - ICML '05 · doi:10.1145/1102351.1102430 · Cited by 1,884
Self-citationCertifying and Removing Disparate Impact
Feldman, M., Friedler, S. A., Moeller, J., Scheidegger, C., & Venkatasubramanian, S. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2783311 · Cited by 1,827
Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission
Caruana, R., Lou, Y., Gehrke, J., Koch, P., Sturm, M., & Elhadad, N. (2015) · Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/2783258.2788613 · Cited by 1,685
Self-citationA Reductions Approach to Fair Classification
Agarwal, A., Beygelzimer, A., Dudík, M., Langford, J., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1803.02453 · Cited by 1,288
Algorithmic Decision Making and the Cost of Fairness
Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017) · Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining · doi:10.1145/3097983.3098095 · Cited by 1,149
Inherent trade-offs in the fair determination of risk scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2017) · Schloss Dagstuhl – Leibniz-Zentrum für Informatik · doi:10.4230/lipics.itcs.2017.43 · Cited by 943
Inherent Trade-Offs in the Fair Determination of Risk Scores
Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016) · arXiv · doi:10.48550/arxiv.1609.05807 · Cited by 941
Datta, A., Sen, S., & Zick, Y. (2016) · 2016 IEEE Symposium on Security and Privacy (SP) · doi:10.1109/sp.2016.42 · Cited by 728 · cited 1×
“February 2–3, 2018, New Orleans, LA, USA © 2018 Copyright held by the owner/author(s). ACM ISBN 978-1-4503-6012-8/18/02. https://doi.org/10.1145/3278721.3278725 Several approaches have been proposed [1, 2, 13, 17, 20, 33] to audit black-box risk scoring models: remove, permute, or obscure a protected feature, then see…”
BackgroundInfluentialThe Comparison and Evaluation of Forecasters
DeGroot, M. H., & Fienberg, S. E. (1983) · The Statistician · doi:10.2307/2987588 · Cited by 604 · cited 1×
“ch we exploit in Section 2.3.3 to detect differences between the mimic and outcome models. 2.3.2 Calibrating model inputs. Calibration is the process of matching predicted and empirical probabilities [14, 28]. If a risk score is well-calibrated, the relationship between the risk score and empirical probabilities will…”
BackgroundInfluentialRiley, R. D., Ensor, J., Snell, K. I. E., Debray, T. P. A., Altman, D. G., Moons, K. G. M., & Collins, G. S. (2016) · BMJ · doi:10.1136/bmj.i3140 · Cited by 585 · cited 1×
“One possible explanation for any nonlinear relationship is that the risk score was well-calibrated on its original training data, but the audit data has a different distribution (data shift) [29].”
BackgroundAccurate intelligible models with pairwise interactions
Lou, Y., Caruana, R., Gehrke, J., & Hooker, G. (2013) · Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2487575.2487579 · Cited by 564
Self-citationIntelligible models for classification and regression
Lou, Y., Caruana, R., & Gehrke, J. (2012) · Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining · doi:10.1145/2339530.2339556 · Cited by 547 · cited 2×
“a statistical test to determine if the audit data is missing key features used to train the black-box model. 5) An ancillary contribution of this paper is a new confidence interval estimate for iGAM [9, 24, 25], a type of transparent model, to compare two models of this class. 2 AUDIT APPROACH Our goal is to gain insi…”
MethodInfluentialSelf-citationPrecinct or prejudice? Understanding racial disparities in New York City’s stop-and-frisk policy
Goel, S., Rao, J. M., & Shroff, R. (2016) · The Annals of Applied Statistics · doi:10.1214/15-aoas897 · Cited by 285 · cited 1×
“[18] proposed a simple risk scoring model for weapon possession: S = 3× 1PS + 1× 1AS + 1× 1Bulдe , where S is the risk score, PS denotes primary stop circumstance is presence of suspicious object, AS denotes secondary stop circumstance is sight or sound of criminal activity, and Bulдe denotes bulge in clothing [18].”
BackgroundAuditing black-box models for indirect influence
Adler, P., Falk, C., Friedler, S. A., Nix, T., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2018) · Knowledge and Information Systems · doi:10.1007/s10115-017-1116-3 · Cited by 280 · cited 1×
“On the other hand, one could study the actual outcome yO , testing for disparate impact through methods such as training a model to predictyO , removing, permuting, or obscuring a protected feature [1, 11], then retraining the model.”
MethodAuditing Black-Box Models for Indirect Influence
Adler, P., Falk, C., Friedler, S. A., Rybeck, G., Scheidegger, C., Smith, B., & Venkatasubramanian, S. (2016) · 2016 IEEE 16th International Conference on Data Mining (ICDM) · doi:10.1109/icdm.2016.0011 · Cited by 280
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramèr, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2015) · arXiv · doi:10.48550/arxiv.1510.02377 · Cited by 171
FairTest: Discovering Unwarranted Associations in Data-Driven Applications
Tramer, F., Atlidakis, V., Geambasu, R., Hsu, D., Hubaux, J.-P., Humbert, M., Juels, A., & Lin, H. (2017) · 2017 IEEE European Symposium on Security and Privacy (EuroS&P) · doi:10.1109/eurosp.2017.29 · Cited by 171 · cited 1×
“tected subgroup. Chouldechova and GâĂŹSell [11] trained two different outcome models then identified subgroups where the two models differed. Other papers work on a single outcome or risk score model [32, 35]. 2.2 Testing for Missing Features If the audit data is missing features used by the black-box model, the audit…”
ResultInfluentialInherent Trade-Offs in Algorithmic Fairness
Kleinberg, J. (2018) · Abstracts of the 2018 ACM International Conference on Measurement and Modeling of Computer Systems · doi:10.1145/3219617.3219634 · Cited by 125
A peek into the black box: exploring classifiers by randomization
Henelius, A., Puolamäki, K., Boström, H., Asker, L., & Papapetrou, P. (2014) · Data Mining and Knowledge Discovery · doi:10.1007/s10618-014-0368-8 · Cited by 98
Research works printed but not linked: 17
- Julius Adebayo and Lalana Kagal . 2016 . Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop . Julius Adebayo and Lalana Kagal. 2016. Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop .
- Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017.
- Susan Athey , Julie Tibshirani , and Stefan Wager . 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 ( 2017 ). Susan Athey, Julie Tibshirani, and Stefan Wager. 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 (2017). · arXiv
- Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS . Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS .
- Alexandra Chouldechova and Max G'Sell . 2017 . Fairer and more accurate, but for whom? . In FATML Workshop . Alexandra Chouldechova and Max G'Sell. 2017. Fairer and more accurate, but for whom?. In FATML Workshop .
- Trevor J Hastie and Robert J Tibshirani. 199 0. Generalized additive models .CRC press. Trevor J Hastie and Robert J Tibshirani. 1990. Generalized additive models .CRC press.
- Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2014 . Distilling the Knowledge in a Neural Network . In NIPS Deep Learning and Representation Learning Workshop . Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop .
- Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML . Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML .
- Michael P Kim , Amirata Ghorbani , and James Zou . 2018 . Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). Michael P Kim, Amirata Ghorbani, and James Zou. 2018. Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). · arXiv
- Manuel Lingo and Gerhard Winkler . 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation ( 2008 ). Manuel Lingo and Gerhard Winkler. 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation (2008).
- Francisco Louzada , Anderson Ara , and Guilherme B Fernandes . 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science ( 2016 ). Francisco Louzada, Anderson Ara, and Guilherme B Fernandes. 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science (2016).
- Lucas Mentch and Giles Hooker . 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research ( 2016 ). Lucas Mentch and Giles Hooker. 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research (2016).
- 10.1016/j.csda.2008.08.007 · DOI
- Sarah Tan , Rich Caruana , Giles Hooker , and Albert Gordo . 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 ( 2018 ). Sarah Tan, Rich Caruana, Giles Hooker, and Albert Gordo. 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 (2018). · arXiv
- Sandra Wachter , Brent Mittelstadt , and Chris Russell . 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology ( 2018 ). Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology (2018).
- Hao Wang , Berk Ustun , and Flavio P. Calmon . 2018 . On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory . Hao Wang, Berk Ustun, and Flavio P. Calmon. 2018. On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory .
- Zhe Zhang and Daniel B. Neill . 2017 . Identifying Significant Predictive Bias in Classifiers. In FATML Workshop . Zhe Zhang and Daniel B. Neill. 2017. Identifying Significant Predictive Bias in Classifiers. In FATML Workshop .
Cited by
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence
Hassija, V., Chamola, V., Mahapatra, A., Singal, A., Goel, D., Huang, K., Scardapane, S., Spinelli, I., Mahmud, M., & Hussain, A. (2024) · Cognitive Computation · doi:10.1007/s12559-023-10179-8 · Cited by 2,495
“Tan et al. [103]. expanded on the notion of distilling intricate black-box models into transparent ones known as iGAMs through model distillation.”
BackgroundFooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods
Slack, D., Hilgard, S., Jia, E., Singh, S., & Lakkaraju, H. (2020) · Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3375627.3375830 · Cited by 900
Markus, A. F., Kors, J. A., & Rijnbeek, P. R. (2021) · Journal of Biomedical Informatics · doi:10.1016/j.jbi.2020.103655 · Cited by 811
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Vaughan, J. W., & Wallach, H. (2018) · arXiv · doi:10.48550/arxiv.1802.07810 · Cited by 711
Metadata
- Authors
- Tan, S., Caruana, R., Hooker, G., & Lou, Y.
- Venue
- Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
- Publisher
- ACM
- Published
- 2018-12-27
- Pages
- 303-310
- ISSN
- 3065-8365 (ISSN-L)
- OpenAlex
- W4289097366
- Cited by
- 125
- References
- 22
- FWCI
- 3.43
- Type
- conference paper
- Language
- en
- Open access
- closed
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.