CoChat

Model Cards for Model Reporting

Open access
Mitchell, Margaret, Wu, Simone, Zaldivar, Andrew, Barnes, Parker, Vasserman, Lucy, Hutchinson, Ben, Spitzer, Elena, Raji, Inioluwa Deborah, Gebru, Timnit · 2019 · Proceedings of the Conference on Fairness, Accountability, and Transparency

The DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; 8 of the 34 sources its claims cite were checked for support.

Abstract

Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.

Overview

CoChat Summary

TL;DR

The paper proposes “model cards,” short documentation records accompanying trained machine learning models to report intended use, performance characteristics, and disaggregated evaluation across demographic, phenotypic, and intersectional groups. It illustrates the framework with worked examples for a smiling classifier and a toxicity classifier.

Key findings

  • The authors argue there are no standardized documentation procedures for communicating trained ML model performance, especially for high-impact domains such as health care, employment, education, and law enforcement.
  • They propose model card sections covering model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats/recommendations.
  • In the smiling classifier example, all reported metrics at the .5 decision threshold have error types within the range 0.04–0.14, with 95% confidence intervals calculated by bootstrap resampling; the card notes older men have much higher false discovery rate, while men in aggregate have higher false negative rate.
  • In the toxicity example, TOXICITY v. 1 has low performance for several terms, especially “lesbian,” “gay,” and “homosexual,” while TOXICITY v. 5 shows more equitable performance after bias mitigation techniques.

Methods

  • The paper is a framework/proposal paper, not an empirical study of a new model; it recommends model cards and details suggested sections and prompts.
  • It provides two worked examples: a smiling classifier trained and evaluated on the CelebA dataset, using a convolutional neural net pretrained for face recognition and fine-tuned with cross-entropy loss for binary smiling classification.
  • It also provides a model card for Perspective API’s TOXICITY classifier, a convolutional neural network developed by Jigsaw in 2017, evaluated on a synthetic Identity Phrase Templates test set; metrics include FPR, FNR, FDR, FOR for the smiling classifier and Pinned AUC for toxicity.

Limitations

  • The paper states that model cards rely on the integrity of their creators and that it seems unlikely, at least in the near term, that they could be standardized or formalized enough to prevent misleading representations of model results.
  • The paper states that evaluation datasets suitable for disaggregated evaluation are not yet common, and that the synthetic toxicity test data covers only a small set of very specific comments and is not comprehensive.
  • The smiling classifier card states it does not capture race or skin type, uses binary gender classes (male/female), and that an ideal evaluation dataset would additionally include Fitzpatrick skin type, camera details, and environment details; the paper also states future work is needed to study how model information is interpreted and used by different stakeholders.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1145/3287560.3287596

Cites

22 of 22 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Figures

Image 1
From the PDF
Figure 3 Example Model Card for two versions of Perspective API's toxicity detector.
Figure 3 Example Model Card for two versions of Perspective API's toxicity detector.From the PDF

Verification

Claims the paper makes

5 Valid · 1 Not valid · 18 Not checked · 30 Own results and interpretations

13 more claims

Create a free account to see all 54 claims and to verify the 13 references still open.

See all claims — free account
  • Probably not a substantial claim. 34
  • It may state two separate claims as one. 8

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

17 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Citations

Cites

Research works printed but not linked: 9

  1. [1] Avrio AI. 2018. Avrio AI: AI Talent Platform. (2018). https://www.goavrio.com/ -
  2. [5] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research) , Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 77-91. http://proceedings.mlr.press/v81/buolamwini18a.html -
  3. [13] Entelo. 2018. Recruitment Software | Entelo. (2018). https://www.entelo.com/ -
  4. [17] U.S. Food and Drug Administration. 2013. FDA Drug Safety Communication: Risk of next-morning impairment after use of insomnia drugs; FDA requires lower recommended doses for certain drugs containing zolpidem (Ambien, Ambien CR, Edluar, and Zolpimist). (2013). https://web.archive.org/web/20170428150213/ https://www.fda.gov/drugs/drugsafety/ucm352085.htm -
  5. [18] IIHS (Insurance Institute for Highway Safety: Highway Loss Data Institute). 2003. Special Issue: Side Impact Crashworthiness. Status Report 38, 7 (2003). -
  6. [19] Institute for the Future, Omidyar Network's Tech, and Society Solutions Lab. 2018. Ethical OS. (2018). https://ethicalos.org/ -
  7. [32] Jigsaw. 2017. Perspective API. (2017). https://www.perspectiveapi.com/ -
  8. [39] AI Now. 2018. Litigating Algorithms: Challenging Government Use Of Algorithmic Decision Systems. AI Now Institute. -
  9. [43] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. In Proceedings of the 34th International Conference on Machine Learning , Vol. 70. PMLR, Sydney, Australia. -

Cited by

See all 2,681

Metadata

Authors
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T.
Venue
Proceedings of the Conference on Fairness, Accountability, and Transparency
Publisher
ACM
Published
2019-01-29
Pages
220-229
OpenAlex
W2897042519
Cited by
2,681
References
12
Citations by year
2026: 831 · 2025: 446 · 2024: 376 · 2023: 366 · 2022: 243 · 2021: 260 · 2020: 102 · 2019: 54
FWCI
170.95
Type
conference paper
Language
en
Open access
Locations
  • ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
  • Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
  • Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
Keywords
model cards, model transparency, algorithmic fairness, demographic bias, responsible AI, democratization of AI, toxic comment detection, model documentation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (gpt-6-sol).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.