CoChat

Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation

Tan, Sarah, Caruana, Rich, Hooker, Giles, Lou, Yin · 2018 · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
Cited by 125

The DOI matches its Crossref record; 21 of the 21 references it cites were found in a registry or an index.

Abstract

Black-box risk scoring models permeate our lives, yet are typically proprietary or opaque. We propose Distill-and-Compare, an approach to audit such models without probing the black-box model API or pre-defining features to audit. To gain insight into black-box models, we treat them as teachers, training transparent student models to mimic the risk scores assigned by the black-box models. We compare the mimic model trained with distillation to a second, un-distilled transparent model trained on ground truth outcomes, and use differences between the two models to gain insight into the black-box model. We demonstrate the approach on four data sets: COMPAS, Stop-and-Frisk, Chicago Police, and Lending Club. We also propose a statistical test to determine if a data set is missing key features used to train the black-box model. Our test finds that the ProPublica data is likely missing key feature(s) used in COMPAS.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3278721.3278725

Cites

21 of 21 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 21 references

Research works printed but not linked: 17

  1. Julius Adebayo and Lalana Kagal . 2016 . Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop . Julius Adebayo and Lalana Kagal. 2016. Iterative Orthogonal Feature Projection for Diagnosing Bias in Black-Box Models. In FATML Workshop .
  2. Julia Angwin , Jeff Larson , Surya Mattu , and Lauren Kirchner . 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017 . Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016a. How we analyzed the compas recidivism algorithm. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm Accessed May 26, 2017.
  3. Susan Athey , Julie Tibshirani , and Stefan Wager . 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 ( 2017 ). Susan Athey, Julie Tibshirani, and Stefan Wager. 2017. Generalized Random Forests. arXiv preprint arXiv:1610.01271 (2017). · arXiv
  4. Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS . Jimmy Ba and Rich Caruana. 2014. Do Deep Nets Really Need to be Deep?. In NIPS .
  5. Alexandra Chouldechova and Max G'Sell . 2017 . Fairer and more accurate, but for whom? . In FATML Workshop . Alexandra Chouldechova and Max G'Sell. 2017. Fairer and more accurate, but for whom?. In FATML Workshop .
  6. Trevor J Hastie and Robert J Tibshirani. 199 0. Generalized additive models .CRC press. Trevor J Hastie and Robert J Tibshirani. 1990. Generalized additive models .CRC press.
  7. Geoffrey Hinton , Oriol Vinyals , and Jeff Dean . 2014 . Distilling the Knowledge in a Neural Network . In NIPS Deep Learning and Representation Learning Workshop . Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2014. Distilling the Knowledge in a Neural Network. In NIPS Deep Learning and Representation Learning Workshop .
  8. Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML . Michael Kearns Seth Neel Aaron Roth and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In ICML .
  9. Michael P Kim , Amirata Ghorbani , and James Zou . 2018 . Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). Michael P Kim, Amirata Ghorbani, and James Zou. 2018. Multiaccuracy: Black-Box Post-Processing for Fairness in Classification. arXiv preprint arXiv:1805.12317 (2018). · arXiv
  10. Manuel Lingo and Gerhard Winkler . 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation ( 2008 ). Manuel Lingo and Gerhard Winkler. 2008. Discriminatory power-an obsolete validation criterion? Journal of Risk Model Validation (2008).
  11. Francisco Louzada , Anderson Ara , and Guilherme B Fernandes . 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science ( 2016 ). Francisco Louzada, Anderson Ara, and Guilherme B Fernandes. 2016. Classification methods applied to credit scoring: Systematic review and overall comparison. Surveys in Operations Research and Management Science (2016).
  12. Lucas Mentch and Giles Hooker . 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research ( 2016 ). Lucas Mentch and Giles Hooker. 2016. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. The Journal of Machine Learning Research (2016).
  13. 10.1016/j.csda.2008.08.007 · DOI
  14. Sarah Tan , Rich Caruana , Giles Hooker , and Albert Gordo . 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 ( 2018 ). Sarah Tan, Rich Caruana, Giles Hooker, and Albert Gordo. 2018. Transparent Model Distillation. arXiv preprint arXiv:1801.08640 (2018). · arXiv
  15. Sandra Wachter , Brent Mittelstadt , and Chris Russell . 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology ( 2018 ). Sandra Wachter, Brent Mittelstadt, and Chris Russell. 2018. Counterfactual Explanations without Opening the Black Box: Automated Decisions and the GDPR. Harvard Journal of Law & Technology (2018).
  16. Hao Wang , Berk Ustun , and Flavio P. Calmon . 2018 . On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory . Hao Wang, Berk Ustun, and Flavio P. Calmon. 2018. On the Direction of Discrimination: An Information-Theoretic Analysis of Disparate Impact in Machine Learning. In International Symposium on Information Theory .
  17. Zhe Zhang and Daniel B. Neill . 2017 . Identifying Significant Predictive Bias in Classifiers. In FATML Workshop . Zhe Zhang and Daniel B. Neill. 2017. Identifying Significant Predictive Bias in Classifiers. In FATML Workshop .

Cited by

See all 125

Metadata

Authors
Tan, S., Caruana, R., Hooker, G., & Lou, Y.
Venue
Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society
Publisher
ACM
Published
2018-12-27
Pages
303-310
ISSN
3065-8365 (ISSN-L)
OpenAlex
W4289097366
Cited by
125
References
21
FWCI
3.43
Type
conference paper
Language
en
Open access
closed

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation. https://app.cochat.ai/papers/10.1145/3278721.3278725
The paper
Tan, S., Caruana, R., Hooker, G., & Lou, Y. (2018). Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation. Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. https://doi.org/10.1145/3278721.3278725

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.