The DOI matches its Crossref record; 62 of the 64 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 20 sources its claims cite is checked for support yet.
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Open accessAbstract
Deep learning models for natural language processing (NLP) are increasingly adopted and deployed by analysts without formal training in NLP or machine learning (ML). However, the documentation intended to convey the model’s details and appropriate use is tailored primarily to individuals with ML or NLP expertise. To address this gap, we conduct a design inquiry into interactive model cards, which augment traditionally static model cards with affordances for exploring model documentation and interacting with the models themselves. Our investigation consists of an initial conceptual study with experts in ML, NLP, and AI Ethics, followed by a separate evaluative study with non-expert analysts who use ML models in their work. Using a semi-structured interview format coupled with a think-aloud protocol, we collected feedback from a total of 30 participants who engaged with different versions of standard and interactive model cards. Through a thematic analysis of the collected data, we identified several conceptual dimensions that summarize the strengths and limitations of standard and interactive model cards, including: stakeholders; design; guidance; understandability & interpretability; sensemaking & skepticism; and trust & safety. Our findings demonstrate the importance of carefully considered design and interactivity for orienting and supporting non-expert analysts using deep learning models, along with a need for consideration of broader sociotechnical contexts and organizational dynamics. We have also identified design elements, such as language, visual cues, and warnings, among others, that support interactivity and make non-interactive content accessible. We summarize our findings as design guidelines and discuss their implications for a human-centered approach towards AI/ML documentation.
Checked 2026-10-06 · 2 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper designs and evaluates interactive model cards (IMCs) that augment static model cards with interactivity and guidance for non-expert NLP/ML analysts. Across two qualitative studies with 30 participants, IMCs were generally favored over standard model cards for understandability and sharing, though ethics and safety content remained hard to make actionable.
Key findings
- In Study 2, nearly all 20 non-expert analysts (n=18) found the IMC easier to read and understand; n=15 said they would use it in daily work, while n=2 said it was context-dependent, and the IMC was overall favored over the standard model card (SMC).
- With the SMC alone, almost half of Study 2 participants could not correctly interpret that the model performed sentiment analysis: four thought it was a general text classification model, two thought DistilBERT and its fine-tuned version were both sentiment analysis models, one did not answer, and one thought the SST-2 dataset was the model; even three who correctly described it were not confident.
- IMC design elements helped engagement: layout (n=13), language (n=6), ability to add examples (n=15), and visual/interactive elements (n=16); n=16 also suggested improvements, and n=4 reported information overload.
- Study 1 experts surfaced conceptual dimensions including stakeholders, design, guidance, trust and safety, and sensemaking and skepticism, leading to design guidelines DG1–DG4 for interactive model cards.
Methods
- Two-study design inquiry: Study 1 was a conceptual study with 10 experts in ML/AI, NLP, ethics, and auditability using paper-prototype probes of an SMC, Robustness Gym report (RGR), and IMC; Study 2 was an evaluative study with 20 non-expert analysts using a functional IMC prototype compared with a reference SMC.
- Both studies used semi-structured interviews with a think-aloud protocol; Study 1 recorded approximately 11 hours 15 minutes of video, and Study 2 recorded approximately 23 hours 30 minutes; analysis used iterative thematic analysis with grounded theory, with Study 1 themes seeding Study 2 analysis.
- Study 2 recruitment used the User Interviews platform: 176 responses, 117 excluded (89 never used ML/NLP models; 28 had graduate or undergraduate training in ML, statistics, or computer science), leaving 59 candidates split into Type I (n=35, no or infrequent programming) and Type II (n=24, daily or weekly programming), from which 20 participants were recruited evenly; the reference model was DistilBERT fine-tuned on SST-2 for sentiment analysis.
Limitations
- Stated: design choices and the choice of sentiment analysis task likely influenced findings; specific IMC design elements may be refined in future work, though the broader design guidelines and conceptual dimensions are expected to generalize; creating and learning to use IMCs is a heavier burden than text-based model cards.
- Stated/future: future work should explore strategies for reducing and redistributing the work of model documentation.
- Evident from the text: the qualitative samples are small (10 experts and 20 analysts) and not statistically representative; Study 2 found no large response differences between analyst groups; ethics and safety remained difficult, with only four participants saying ethical templates could stimulate discussion and two pushing back against more actionable ethics/safety content.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
63 Own result · 32 Own interpretation · 9 Not checked yet · 3 Own method · 2 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, almost half of the participants were not able to provide the correct interpretation of what the model did using the SMC alone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with participants echoing the concerns that were previously articulated by the experts in Section 3.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supporting interpretability is complex but nearly all (n=7) participants indicated that examples could provide some useful guidance.”
hedgedn = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A few participants (n=4) expressed that text, visuals, and interaction options led to information overload.”
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reference model card (Section 3.1), here acting as the SMC, and the IMC each had features that were preferred by both types of analysts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to the prior study, non-expert analysts provided greater depth towards the themes of stakeholders ; design considerations ; and sensemaking & skepticism in particular.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants indicated that interactivity added substantially to the SCM because it allowed participants to 'play with the data and see the results [in a way that] that feels super intuitive instead of explaining everything' [A1-09].”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One of the most salient implications of the IMC design was that it helped participants accurately understand what the model was and what it did.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants indicated that various stakeholders would need help making sense of the information in the model card.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even three participants who did correctly describe the model were not confident in their interpretation”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 35
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the addition of guiding elements, suggested in the previous study (Table 2), were well received and did help participants understand and interpret the model card information.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our conceptual study emphasized the importance of information design within both standard and interactive model cards.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants indicated that for these issues and overall, greater attention to information, visual, and interaction design would improve trust and safety.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several (n=4) participants expressed that it could be beneficial to add sensitive populations to the disaggregated model performance, but they suggested caution since the definition of these groups was culturally and geographically contextual.”
hedgedn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants suggested that integrating the model's code and files, present in the SMC, would be valuable for the IMC.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Warnings, prompts, and summaries are all examples of guidance that participants suggested could help guide analysts in making sense of model cards.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When asked, nearly all participants (n=15) indicated that the IMC was something they would use in their daily work, while some (n=2) indicated it was context-dependent.”
hedgedn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“one participant initially thought the model performed well for a particular subpopulation, then saw the small sample size and revised their interpretation.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the interactivity would enable people to 'elicit more questions' [E04] that allowed them to form and test their mental models about the data and better 'see the things that model lacks' [E05].”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants articulated that several components of the IMC helped them engage with the model and data more effectively by clarifying and prioritizing information, particularly: layout of information (n=13); language (n=6); ability to add examples (n=15); and visual and interactive elements (n=16).”
plaindirectn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“experts saw the opportunity to use interaction as a way to help non-expert analysts develop their conceptual understanding of the NLP model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three participants voiced the importance of being able to contest the model's results, and wanted to see this ability expanded beyond adding sentences (Section 4.1) to the model's training and testing data or uploading their own dataset.”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasingly, such statements are seen as a 'cop-out because it just kind of says 'there is some bias somewhere' which is not entirely helpful.' [E06]”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nearly all (n=16) participants also made suggestions for improvements to the IMC design. These included changes to elements of the interface that were confusing (n=5), the layout (n=6), and types of guidance (n=1) provided.”
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they emphasized the need to prominently display the total size of these subpopulations (and others) to support valid interpretation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We found that our approach to guidance, shareability, and traceability begins to address these needs, and we surfaced additional relevant implications for design.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: The DistilBERT SST-2 sentiment analysis task is representative of NLP model development regimens for other tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants expressed doubt that a non-expert analyst would understand the model card or know how to act on the information it contained.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, a sense of focus/overwhelm varied among participants, and one participant articulated that the IMC contained 'too much information and is very cluttered' [E04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two participants suggested stating explicitly whether bias assessments had been conducted, and one suggested raising a warning if such bias analyses are not run.”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four participants stated that a summary of the model's purpose (i.e., 'this is a sentiment analysis model') was needed and that it would improve the readability of the model card.”
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, having an effective default example to prompt end-users was also important [E04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the IMC included templates to examine ethically challenging examples (Section 4.1), only four participants expressed that these features could stimulate discussion.”
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the IMC also provided template sentences to promote ethical thinking, participants largely favored their own examples.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, three participants expressed that IMCs, if they could be easy to produce, would lower their existing model documentation burden.”
hedgedconditionaln = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our research supports prior findings that these concepts are challenging to integrate into decision-making [48, 60], and we identified some cynicism toward ethics statements, as well as assertions that ethical questions require contextual, situated examination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Others felt that topics of safety and ethics were too abstract, with one participant stating that they 'wouldn't worry too much about that to be honest, I guess it depends on the type of analysis you're doing.' [A2-05].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, the reference model card contained information and actions to obtain the model code and launch it - something that was absent in the IMC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unexpectedly, two participants pushed back against the inclusion of more actionable ethical and safety content, articulating concerns that too much guidance could be detrimental and provide a false sense of security.”
plaindirectn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant expressed doubt that any model documentation could really give an adequate picture of ethical and safety risks, emphasizing that ' 'this [model card] would not be my only data point or else, you know, that's a very quick way to lose your job' [A1-04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, to be sure that this content is effectively understood by a wide audience, the hierarchy and design of this content is equally important.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Video transcripts were automatically generated, including speaker identification, and were verified for accuracy by the authors.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This depth allowed us to expand on the subthemes from Table 2 and broaden our conceptions of how model cards might be used.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had exposure to documentation for ML/AI models in some form, but only six were familiar with model cards and two with RGRs.”
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Robustness Gym helps model developers report on model behavior through the generation of robustness reports (RGRs) that summarize model performance on various data slices [23].”
What the sources say
- Not checked yetcites [23]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Clear guidance can support non-experts in understanding and knowing how to act on the information.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional guidance will be necessary to support interpretability of the outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: SMCs have been primarily tailored toward developers and machine learning experts.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that this productive skepticism would make ideas of intended and unintended uses more intuitive than simply being told what they are.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Not all readers of a model card will want to interact with it.
None of the sources cited for this claim have been checked yet.
In the paper
“Prior work has shown that non-expert analysts benefit from interacting with machine learning models and their data [1, 19, 59].”
What the sources say
- Not checked yetcites [1]
- Not checked yetcites [19] source
- Not checked yetcites [59]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, we found that scaffolding these interactions to support sensemaking requires clear and actionable guidance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1) Interactive model cards are a bridge to further analysis, not a substitute.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also recognize that creating and learning how to use IMCs is a heavier burden than for text-based model cards.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited participants are representative of the future data workforce.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For model cards to be adopted and impact organizational processes, they must support knowledge sharing and negotiation across stakeholders with diverse backgrounds and perspectives.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that with thoughtfully designed interaction, model cards can be a first step for data workers to explore how trustworthy a model is and whether it might meet their needs, serving as a guide to deeper interrogation when desired.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our findings indicate, unsurprisingly, that model documentation can spark ethical thinking, but will never substitute for it.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Design choices and the choice of sentiment analysis tasks (Section 3.1) likely influenced our findings.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For individual stakeholders, the most effective form for interactivity in model cards will be situated and contextual.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our analysis reinforces that considerations of ethics are personal and contextual, so they are unlikely to be addressed with a single approach.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Their feedback validates many of our design choices, but also offers avenues for further improvements.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we believe that the broader design guidelines (Section 3.4) and conceptual dimensions that we surfaced (Tables 2, 4, and 5) are robust and will generalize to other tasks and contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: The recruited participants are representative of the future data workforce.
The paper relies on this without support.
Model cards and similar documentation rely on the developer to accurately and clearly report on the model and its performance.
“However, these model cards and similar forms of documentation rely on the developer to accurately and clearly report on the model and its performance.”
The paper relies on this without support.
Many important details, such as unintended uses or disaggregated model performance, are omitted from model cards.
“Often, this process is labor-intensive and many important details, such as unintended uses or disaggregated model performance, are omitted.”
The paper relies on this without support.
The DistilBERT SST-2 sentiment analysis task is representative of NLP model development regimens for other tasks.
“We chose to focus on a sentiment analysis task because it is widely known and easily understood, while still being representative of NLP model development regimens for other tasks.”
The paper relies on this without support.
The design probes (SMC, RGR, IMC) adequately represent the respective documentation types.
“We developed design probes [70] of an SMC, RGR, and IMC. For the SMC, we used the all the text from the reference model card, with some exceptions, laid out according to the specifications in Mitchell et al. [44].”
The paper relies on this without support.
The model card format is text-based and concise, making it both broadly accessible and applicable across model types.
“The format they propose is text-based and concise, making it both broadly accessible and applicable across model types.”
The paper relies on this without support.
Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.
“Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.”
The paper relies on this without support.
SMCs have been primarily tailored toward developers and machine learning experts.
“SMCs have been primarily tailored toward developers and machine learning experts”
The paper relies on this without support.
Not all readers of a model card will want to interact with it.
“Not all readers of a model card will want to interact with it.”
The paper relies on this without support.
Technology that enables low or no code data analysis is lowering barriers to developing and deploying deep learning models.
“Technology that enables low or no code data analysis is lowering barriers to developing and deploying deep learning models.”
The paper relies on this without support.
The data workers taking on AI/ML development do not have formal education in the theoretical and technical underpinnings of these systems.
“Increasingly, the data workers taking on AI/ML development do not have formal education in the theoretical and technical underpinnings of these systems.”
The paper relies on this without support.
The recruited participants are representative of the future data workforce.
“We recruited and explored the perspectives of people who we believe are representative of the future data workforce.”
The paper relies on this without support.
Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
“Model cards are intended to surface, amongother things, ethical and safety implications - a topic that is especially pressing for deep learning models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Model cards represent the field's standard for disclosing and documenting key facts about the architecture, training and evaluation of the model [14, 40].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [14]
Citations
Cites
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 12,930
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Research works printed but not linked: 14
- Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Mag. 35(2014), 105–120.
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”.
- Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of experimental psychology. General 144 1 (2015), 114–26.
- Alexandra Giannopoulou. 2020. Algorithmic Systems: the Consent is in the Detail?Internet Policy Review 9, 1 (2020).
- Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv preprint arXiv:2101.04840(2021). · arXiv
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black Box Models. ACM computing surveys (CSUR) 51, 5 (2018), 1–42.
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory Framework for Algorithmic Governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–35.
- Andreas Madsen Siva Reddy and Sarath Chandar. 2021. Post-hoc Interpretability for Neural NLP: A Survey. arXiv preprint arXiv:2108.04840(2021). · arXiv
- Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. 2020. Interpretable Machine Learning–A Brief History, State-of-the-Art and Challenges. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 417–431.
- Michael Muller. 2014. Curiosity, Creativity, and Surprise as Analytic Tools: Grounded Theory Method. In Ways of Knowing in HCI. Springer, 25–48.
- Radim Rehurek and Petr Sojka. 2011. Gensim–python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic 3, 2(2011).
- Salome Viljoen. 2021. A Relational Theory of Data Governance. The Yale Law Journal 131, 573 (2021).
- Meredith Whittaker, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaziunas, Varoon Mathur, Sarah Mysers West, Rashida Richardson, Jason Schultz, and Oscar Schwartz. 2018. AI Now Report 2018. AI Now Institute at New York University New York.
- Jilei Yang, Diana Negoescu, and Parvez Ahammad. 2021. Intellige: A User-Facing Model Explainer for Narrative Explanations. arXiv preprint arXiv:2105.12941(2021). · arXiv
Cited by
Sorted by citations
Items 101–102 of 102
The AIMS-Templates: Documenting the multidimensional sustainability of AI systems
State, L., Züger, T., & Winter, L. (2026) · International Conference on Information Technology for Social Good (GoodIT) · doi:10.1145/3794786.3830766
Lu, H., Han, J., Huang, Y., Liu, J., Bao, L., Han, S., Lo, D., & Deng, S. (2026) · Journal of Software Evolution and Process · doi:10.1002/smr.70170 · Cited by 1
Metadata
- Authors
- Crisan, A., Drouhard, M., Vig, J., & Rajani, N.
- Affiliations
- Tableau Software (United States) (US)
- Salesforce (United States) (US)
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 427-439
- OpenAlex
- W4229447062
- Cited by
- 101
- References
- 56
- Citations by year
- 2026: 24 · 2025: 31 · 2024: 22 · 2023: 23 · 2022: 1
- FWCI
- 10.64
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model documentation, human-centered design, non-expert users, deep learning models, natural language processing, AI ethics, model interpretability, trust and safety, design guidelines, stakeholder perspectives, organizational dynamics, socio-technical context
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
