CoChat
Sign Up
PreprintCited by 1,165Explainable Artificial Intelligence (XAI)Statistical Methods and InferenceAdvanced Causal Inference Techniques

All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously

Open access
Fisher, Aaron, Rudin, Cynthia, Dominici, Francesca·2018·arXiv
DOI 10.48550/arxiv.1801.01489arXiv 1801.01489
Open this paper in CoChat — free accountOpen PDF

Abstract

Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})=\mathbf{x}^{T}β$ with a fixed coefficient vector $β$) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.

Integrity signals

Registry

DataCite · 10.48550/arxiv.1801.01489

Cites

8 of 8 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Model Class Reliance can provide a more comprehensive understanding of relationships between variables and predictions than approaches that study individual models.

    “By studying an entire class of models simultaneously, the approach is able to provide a more comprehensive understanding of the relationships between variables and predictions compared to approaches that study individual models.”

    From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [109]

  • Model Class Reliance may be computationally expensive because it requires training and evaluating multiple models.

    “However, one potential disadvantage of this approach is that it may be computationally expensive, as it requires training and evaluating multiple models.”

    From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [109]

  • Model Class Reliance can help identify variables that are consistently important across different models, providing more robust and trustworthy explanations.

    “This approach can also help to identify variables that are consistently important across different models, providing more robust and trustworthy explanations.”

    From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [109]

  • Model Class Reliance may struggle with complex models, especially those with many variables or non-linear relationships.

    “Another disadvantage is that the approach may struggle with handling complex models, especially those with many variables or non-linear relationships.”

    From Interpreting Black-Box Models: A Review on Explainable Artificial Intelligence (2024) · cites this paper as [109]

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Fisher, A., Rudin, C., & Dominici, F.
Affiliations
  • Duke University (US)
  • Harvard University (US)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2018
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.1801.01489
arXiv
1801.01489
OpenAlex
W2969476445
Cited by
1,165
References
8
Citations by year
2026: 17 · 2025: 244 · 2024: 226 · 2023: 241 · 2022: 185 · 2021: 160 · 2020: 71 · 2019: 18 · 2018: 1 · 2017: 1 · 2016: 1
Type
posted content
Language
en
Open access
Licence
arxiv.org/licenses/nonexclusive-distrib/1.0/
Locations
  • arXiv (Cornell University) (publishedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Mathematics › Statistics and Probability › Statistical Methods and Inference
  • Physical Sciences › Mathematics › Statistics and Probability › Advanced Causal Inference Techniques
Keywords
feature importance, permutation importance, random forest, U-statistics, probability bounds, recidivism prediction, criminal records, model interpretability

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review