Model Cards for Model Reporting
Open accessThe DOI matches its Crossref record; 22 of the 22 references it cites were found in a registry or an index; 8 of the 34 sources its claims cite were checked for support.
Abstract
Trained machine learning models are increasingly used to perform high-impact tasks in areas such as law enforcement, medicine, education, and employment. In order to clarify the intended use cases of machine learning models and minimize their usage in contexts for which they are not well suited, we recommend that released models be accompanied by documentation detailing their performance characteristics. In this paper, we propose a framework that we call model cards, to encourage such transparent model reporting. Model cards are short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, or phenotypic groups (e.g., race, geographic location, sex, Fitzpatrick skin type [15]) and intersectional groups (e.g., age and race, or sex and Fitzpatrick skin type) that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information. While we focus primarily on human-centered machine learning models in the application fields of computer vision and natural language processing, this framework can be used to document any trained machine learning model. To solidify the concept, we provide cards for two supervised models: One trained to detect smiling faces in images, and one trained to detect toxic comments in text. We propose model cards as a step towards the responsible democratization of machine learning and related artificial intelligence technology, increasing transparency into how well artificial intelligence technology works. We hope this work encourages those releasing trained machine learning models to accompany model releases with similar detailed evaluation numbers and other relevant documentation.
Overview
CoChat Summary
TL;DR
The paper proposes “model cards,” short documentation records accompanying trained machine learning models to report intended use, performance characteristics, and disaggregated evaluation across demographic, phenotypic, and intersectional groups. It illustrates the framework with worked examples for a smiling classifier and a toxicity classifier.
Key findings
- The authors argue there are no standardized documentation procedures for communicating trained ML model performance, especially for high-impact domains such as health care, employment, education, and law enforcement.
- They propose model card sections covering model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats/recommendations.
- In the smiling classifier example, all reported metrics at the .5 decision threshold have error types within the range 0.04–0.14, with 95% confidence intervals calculated by bootstrap resampling; the card notes older men have much higher false discovery rate, while men in aggregate have higher false negative rate.
- In the toxicity example, TOXICITY v. 1 has low performance for several terms, especially “lesbian,” “gay,” and “homosexual,” while TOXICITY v. 5 shows more equitable performance after bias mitigation techniques.
Methods
- The paper is a framework/proposal paper, not an empirical study of a new model; it recommends model cards and details suggested sections and prompts.
- It provides two worked examples: a smiling classifier trained and evaluated on the CelebA dataset, using a convolutional neural net pretrained for face recognition and fine-tuned with cross-entropy loss for binary smiling classification.
- It also provides a model card for Perspective API’s TOXICITY classifier, a convolutional neural network developed by Jigsaw in 2017, evaluated on a synthetic Identity Phrase Templates test set; metrics include FPR, FNR, FDR, FOR for the smiling classifier and Pinned AUC for toxicity.
Limitations
- The paper states that model cards rely on the integrity of their creators and that it seems unlikely, at least in the near term, that they could be standardized or formalized enough to prevent misleading representations of model results.
- The paper states that evaluation datasets suitable for disaggregated evaluation are not yet common, and that the synthetic toxicity test data covers only a small set of very specific comments and is not comprehensive.
- The smiling classifier card states it does not capture race or skin type, uses binary gender classes (male/female), and that an ideal evaluation dataset would additionally include Fitzpatrick skin type, camera details, and environment details; the paper also states future work is needed to study how model information is interpreted and used by different stakeholders.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Figures
Verification
Claims the paper makes
5 Valid · 1 Not valid · 18 Not checked · 30 Own results and interpretations
The cited sources support only part of this claim — read each source’s verdict below.
In the paper
“Synthetic data is valuable here because [11] shows that real data often has disproportionate amounts of toxicity directed at specific groups.”
What the sources say
- Partly supportedcites [11] [11] Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. 2018. Measuring and Mitigating Unintended Bias in Text Classification. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society (2018). - sourceExperimental — accuracy not yet measured
The 'Quantifying bias in dataset' section (page unavailable) reports that certain identity terms appear disproportionately in toxic comments in the authors’ training data; for example, 'gay' appears in 3% of toxic comments but 0.5% of comments overall. It measures comments containing those terms, not whether toxicity is directed at the groups, and does not establish how often this pattern occurs across real datasets generally. (LLM, conf=0.85)
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Partly supported
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For example, the false discovery rate on older men is much higher than that for other groups.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“TOXICITY v. 1 has low performance for several terms, especially 'lesbian', 'gay', and 'homosexual'.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
None of the sources cited for this claim have been checked yet.
In the paper
“she collaborated with other researchers to demonstrate the disproportionate errors of computer vision systems on historically marginalized groups in the United States, such as darker-skinned women [5, 41].”
What the sources say
- Not checked yetcites [5] [5] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research) , Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 77-91. http://proceedings.mlr.press/v81/buolamwini18a.html -
- Not checked yetcites [41] [41] Inioluwa Raji. 2018. Black Panther Face Scorecard: Wakandans Under the Coded Gaze of AI. (2018). -
boostedstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
In the paper
“after MIT Media Lab graduate student Joy Buolamwini found that commercial face recognition systems failed to detect her face [4]”
What the sources say
- Not checked yetcites [4] [4] Joy Buolamwini. 2016. How I'm fighting Bias in Algorithms. (2016). https://www.ted.com/talks/joy\_buolamwini\_how\_i\_m\_fighting\_bias\_in\_ algorithms#t-63664 -
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
13 more claims
Create a free account to see all 54 claims and to verify the 13 references still open.
- Probably not a substantial claim. 34
- It may state two separate claims as one. 8
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Model cards represent the field's standard for disclosing and documenting key facts about the architecture, training and evaluation of the model [14, 40].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [40]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Model cards (SMCs) [44] have been foundational in standardizing documentation for AI/ML model performance and characteristics, along with risks and unintended uses.”
From Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022) · cites this paper as [44]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“A second phase saw a more technical approach from the computer science community focusing on fairness, accountability and transparency as an engineering 'ethicalby-design' problem-solving exercise [20][21][22].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“These are either materials for use in design teams in workshop style events [76,77], tools for producing documentation of the design, build and test process [21,68], or technical tools for testing models, protecting privacy and security, testing for bias [69,93,94], or tracking provenance of data [95].”
From Putting AI ethics to work: are the tools fit for purpose? (2022) · cites this paper as [21]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“There is some work on fairness relating to AI services, though this tends to focus on either exposing or 'auditing' certain AI services [22,38,126,133], or devising tools, methods and interventions relevant for those building (rather than using) such services [12,96,106,128].”
From Out of Context: Investigating the Bias and Fairness Concerns of “Artificial Intelligence as a Service” (2023) · cites this paper as [106]
Experimental — accuracy not yet measured
17 more claims about this paper
Create a free account to see every claim other papers make about this one.
Citations
Cites
BLEU: a method for automatic evaluation of machine translation
Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2001) · Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL '02 · doi:10.3115/1073083.1073135 · Cited by 21,344 · cited 1×
“translation system could compare BLEU scores [40] across demographic groups, and a model card for a speech recognition system could compare word-error rates.”
BackgroundCrenshaw, K. (2018) · Feminist Legal Theory · doi:10.4324/9780429500480-5 · Cited by 9,510
Deep Learning Face Attributes in the Wild
Liu, Z., Luo, P., Wang, X., & Tang, X. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.425 · Cited by 7,747 · cited 4×
“ms [5, 26] that report details of the datasets used to train and test machine learning models. We provide two example model cards in Section 5: A smiling detection model trained on the CelebA dataset [33] (Figure 2), and a public toxicity detection model, Perspective API [29] (Figure 3). Where Datasheets highlight cha…”
BackgroundMethodInfluentialThe validity and practicality of sun-reactive skin types I through VI
Fitzpatrick, T. B. (1988) · Archives of Dermatology · doi:10.1001/archderm.1988.01670060015008 · Cited by 4,339
Differential Privacy: A Survey of Results
Dwork, C. (2008) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-79228-4_1 · Cited by 3,542 · cited 2×
“As such, collaboration with policy, privacy, and legal experts is necessary in order to ascertain which groups may be responsibly inferred, and how that information should be stored and accessed (for example, using differential privacy [12]).”
BackgroundCollins, G. S., Reitsma, J. B., Altman, D. G., & Moons, K. G. M. (2015) · Annals of Internal Medicine · doi:10.7326/m14-0697 · Cited by 3,267
Equality of Opportunity in Supervised Learning
Hardt, M., Price, E., & Srebro, N. (2016) · arXiv · doi:10.48550/arxiv.1610.02413 · Cited by 2,536 · cited 2×
“of Opportunity, and equal false negative and false positive rates across groups is equivalent to fulfilling Equality of Odds [26].”
BackgroundGebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Self-citationA Probabilistic Interpretation of Precision, Recall and F-Score, with Implication for Evaluation
Goutte, C., & Gaussier, E. (2005) · Lecture Notes in Computer Science · doi:10.1007/978-3-540-31865-1_25 · Cited by 2,285 · cited 1×
“Confidence intervals for metrics derived from confusion matrices can be calculated by treating the matrices as probabilistic models of system performance [24].”
MethodFair prediction with disparate impact: a study of bias in recidivism prediction instruments
Chouldechova, A. (2017) · Big Data · doi:10.1089/big.2016.0047 · Cited by 2,159 · cited 1×
“or different errors that can be calculated from the confusion matrix for binary classification systems. • These also correspond to metrics in recent definitions of “fairness” in machine learning (cf. [8, 25]), where parity across subgroups for different metrics correspond to different fairness criteria. • 95% confiden…”
BackgroundKim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., & Sayres, R. (2017) · arXiv · doi:10.48550/arxiv.1711.11279 · Cited by 1,240
Fairness definitions explained
Verma, S., & Rubin, J. (2018) · Proceedings of the International Workshop on Software Fairness · doi:10.1145/3194770.3194776 · Cited by 1,124 · cited 2×
“FalseOmissionRate, which measure the fraction of negative (not smiling) and positive (smiling) predictions that are incorrectly predicted to be positive and negative, respectively, are also reported. [48] • Together, these four metrics provide values for different errors that can be calculated from the confusion matri…”
BackgroundInfluentialPrediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2018) · arXiv · doi:10.48550/arxiv.1811.07867 · Cited by 687
Algorithmic Fairness: Choices, Assumptions, and Definitions
Mitchell, S., Potash, E., Barocas, S., D'Amour, A., & Lum, K. (2021) · Annual Review of Statistics and Its Application · doi:10.1146/annurev-statistics-042720-125902 · Cited by 687
Measuring and Mitigating Unintended Bias in Text Classification
Dixon, L., Li, J., Sorensen, J., Thain, N., & Vasserman, L. (2018) · Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3278721.3278729 · Cited by 674 · cited 7×
“A model card could even extend beyond these summary statistics to reveal other measures of differences between distributions such as cross entropy, perplexity, KL divergence and pinned area under the curve (pinned AUC) [13].”
BackgroundMethodInfluentialSelf-citationFace Recognition Performance: Role of Demographic Information
Klare, B. F., Burge, M. J., Klontz, J. C., Vorder Bruegge, R. W., & Jain, A. K. (2012) · IEEE Transactions on Information Forensics and Security · doi:10.1109/tifs.2012.2214212 · Cited by 516 · cited 1×
“ially problematic when models are used in applications that have serious impacts on people’s lives, such as in health care [14, 42, 44], employment [1, 13, 29], education [23, 45] and law enforcement [2, 7, 20, 34]. Researchers have discovered systematic biases in commercial machine learning models used for face detec…”
BackgroundThe Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Holland, S., Hosny, A., Newman, S., Joseph, J., & Chmielinski, K. (2018) · arXiv · doi:10.48550/arxiv.1805.03677 · Cited by 175
Did the Model Understand the Question?
Mudrakarta, P. K., Taly, A., Sundararajan, M., & Dhamdhere, K. (2018) · Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) · doi:10.18653/v1/p18-1176 · Cited by 166 · cited 1×
“Additional details may include, for example, interpretability approaches, such as saliency maps, TCAV [33], and Path-Integrated Gradients [38, 43]); stakeholder-relevant explanations (e.”
BackgroundIncreasing Trust in AI Services through Supplier's Declarations of Conformity
Hind, M., Mehta, S., Mojsilovic, A., et al. (2018) · arXiv.org · doi:10.48550/arxiv.1808.07261 · Cited by 134
Li, D., Hu, S. C., Lin, L.-S., & Yeh, C.-W. (2017) · PLoS ONE · doi:10.1371/journal.pone.0181853 · Cited by 40 · cited 1×
“In some of these situations, synthetically generated datasets may provide representation for use cases that would otherwise go unevaluated [32].”
Background(2018) · Cited by 17
Hybrid sensing face detection and registration for low-light and unconstrained conditions
Zhou, M., Lin, H., Young, S. S., & Yu, J. (2018) · Applied Optics · doi:10.1364/ao.57.000069 · Cited by 12 · cited 1×
“rther factor affecting model performance is the environment in which it is deployed. For example, face detection systems are often less accurate under low lighting conditions or when the air is humid [45]. Specifications across different lighting and moisture conditions would help users understand the impacts of these…”
BackgroundInfluential
Research works printed but not linked: 9
- [1] Avrio AI. 2018. Avrio AI: AI Talent Platform. (2018). https://www.goavrio.com/ -
- [5] Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research) , Sorelle A. Friedler and Christo Wilson (Eds.), Vol. 81. PMLR, New York, NY, USA, 77-91. http://proceedings.mlr.press/v81/buolamwini18a.html -
- [13] Entelo. 2018. Recruitment Software | Entelo. (2018). https://www.entelo.com/ -
- [17] U.S. Food and Drug Administration. 2013. FDA Drug Safety Communication: Risk of next-morning impairment after use of insomnia drugs; FDA requires lower recommended doses for certain drugs containing zolpidem (Ambien, Ambien CR, Edluar, and Zolpimist). (2013). https://web.archive.org/web/20170428150213/ https://www.fda.gov/drugs/drugsafety/ucm352085.htm -
- [18] IIHS (Insurance Institute for Highway Safety: Highway Loss Data Institute). 2003. Special Issue: Side Impact Crashworthiness. Status Report 38, 7 (2003). -
- [19] Institute for the Future, Omidyar Network's Tech, and Society Solutions Lab. 2018. Ethical OS. (2018). https://ethicalos.org/ -
- [32] Jigsaw. 2017. Perspective API. (2017). https://www.perspectiveapi.com/ -
- [39] AI Now. 2018. Litigating Algorithms: Challenging Government Use Of Algorithmic Decision Systems. AI Now Institute. -
- [43] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. In Proceedings of the 34th International Conference on Machine Learning , Vol. 70. PMLR, Sydney, Australia. -
Cited by
Arrieta, A. B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., García, S., Gil-López, S., Molina, D., Benjamins, R., et al. (2019) · arXiv · doi:10.48550/arxiv.1910.10045 · Cited by 11,321
“XAI tools and fairness tools fall in this category, as well as other recent proposals such as model cards [426] .”
Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A. C., Lo, W.-Y., et al. (2023) · 2023 IEEE/CVF International Conference on Computer Vision (ICCV) · doi:10.1109/iccv51070.2023.00371 · Cited by 11,953
“We provide a Model Card following [75] 4) The data is more geographically diverse than its predecessors, and we hope it will bring the community one step closer to creating fairer and more equitable models.”
BackgroundBarredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
“Additional model-specific metadata can be provided via a model card (Mitchell et al., 2018) that describes properties of its training, a citation to the work, datasets used during pretraining, and any caveats about known biases in the model and its predictions.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
“While documentation allows for potential accountability [13, 52, 86], undocumented training data perpetuates harm without recourse.”
Self-citation
Metadata
- Authors
- Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T.
- Venue
- Proceedings of the Conference on Fairness, Accountability, and Transparency
- Publisher
- ACM
- Published
- 2019-01-29
- Pages
- 220-229
- OpenAlex
- W2897042519
- Semantic Scholar
- 7365f887c938ca21a6adbef08b5a520ebbd4638f
- Cited by
- 2,681
- References
- 12
- Citations by year
- 2026: 831 · 2025: 446 · 2024: 376 · 2023: 366 · 2022: 243 · 2021: 260 · 2020: 102 · 2019: 54
- FWCI
- 170.95
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Face recognition and analysis
- Keywords
- model cards, model transparency, algorithmic fairness, demographic bias, responsible AI, democratization of AI, toxic comment detection, model documentation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-07, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-07 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (gpt-6-sol).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.

