The DOI matches its Crossref record; 62 of the 64 references it cites were found in a registry or an index, and 2 are not checked yet; none of the 20 sources its claims cite is checked for support yet.
Interactive Model Cards: A Human-Centered Approach to Model Documentation
Open accessAbstract
Deep learning models for natural language processing (NLP) are increasingly adopted and deployed by analysts without formal training in NLP or machine learning (ML). However, the documentation intended to convey the model’s details and appropriate use is tailored primarily to individuals with ML or NLP expertise. To address this gap, we conduct a design inquiry into interactive model cards, which augment traditionally static model cards with affordances for exploring model documentation and interacting with the models themselves. Our investigation consists of an initial conceptual study with experts in ML, NLP, and AI Ethics, followed by a separate evaluative study with non-expert analysts who use ML models in their work. Using a semi-structured interview format coupled with a think-aloud protocol, we collected feedback from a total of 30 participants who engaged with different versions of standard and interactive model cards. Through a thematic analysis of the collected data, we identified several conceptual dimensions that summarize the strengths and limitations of standard and interactive model cards, including: stakeholders; design; guidance; understandability & interpretability; sensemaking & skepticism; and trust & safety. Our findings demonstrate the importance of carefully considered design and interactivity for orienting and supporting non-expert analysts using deep learning models, along with a need for consideration of broader sociotechnical contexts and organizational dynamics. We have also identified design elements, such as language, visual cues, and warnings, among others, that support interactivity and make non-interactive content accessible. We summarize our findings as design guidelines and discuss their implications for a human-centered approach towards AI/ML documentation.
Checked 2026-10-06 · 2 references not checked yet
Overview
AI snapshot
Written from the paper’s full text.
TL;DR
The paper designs and evaluates interactive model cards (IMCs) that augment static model cards with interactivity and guidance for non-expert NLP/ML analysts. Across two qualitative studies with 30 participants, IMCs were generally favored over standard model cards for understandability and sharing, though ethics and safety content remained hard to make actionable.
Key findings
- In Study 2, nearly all 20 non-expert analysts (n=18) found the IMC easier to read and understand; n=15 said they would use it in daily work, while n=2 said it was context-dependent, and the IMC was overall favored over the standard model card (SMC).
- With the SMC alone, almost half of Study 2 participants could not correctly interpret that the model performed sentiment analysis: four thought it was a general text classification model, two thought DistilBERT and its fine-tuned version were both sentiment analysis models, one did not answer, and one thought the SST-2 dataset was the model; even three who correctly described it were not confident.
- IMC design elements helped engagement: layout (n=13), language (n=6), ability to add examples (n=15), and visual/interactive elements (n=16); n=16 also suggested improvements, and n=4 reported information overload.
- Study 1 experts surfaced conceptual dimensions including stakeholders, design, guidance, trust and safety, and sensemaking and skepticism, leading to design guidelines DG1–DG4 for interactive model cards.
Methods
- Two-study design inquiry: Study 1 was a conceptual study with 10 experts in ML/AI, NLP, ethics, and auditability using paper-prototype probes of an SMC, Robustness Gym report (RGR), and IMC; Study 2 was an evaluative study with 20 non-expert analysts using a functional IMC prototype compared with a reference SMC.
- Both studies used semi-structured interviews with a think-aloud protocol; Study 1 recorded approximately 11 hours 15 minutes of video, and Study 2 recorded approximately 23 hours 30 minutes; analysis used iterative thematic analysis with grounded theory, with Study 1 themes seeding Study 2 analysis.
- Study 2 recruitment used the User Interviews platform: 176 responses, 117 excluded (89 never used ML/NLP models; 28 had graduate or undergraduate training in ML, statistics, or computer science), leaving 59 candidates split into Type I (n=35, no or infrequent programming) and Type II (n=24, daily or weekly programming), from which 20 participants were recruited evenly; the reference model was DistilBERT fine-tuned on SST-2 for sentiment analysis.
Limitations
- Stated: design choices and the choice of sentiment analysis task likely influenced findings; specific IMC design elements may be refined in future work, though the broader design guidelines and conceptual dimensions are expected to generalize; creating and learning to use IMCs is a heavier burden than text-based model cards.
- Stated/future: future work should explore strategies for reducing and redistributing the work of model documentation.
- Evident from the text: the qualitative samples are small (10 experts and 20 analysts) and not statistically representative; Study 2 found no large response differences between analyst groups; ethics and safety remained difficult, with only four participants saying ethical templates could stimulate discussion and two pushing back against more actionable ethics/safety content.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
63 Own result · 32 Own interpretation · 9 Not checked yet · 3 Own method · 2 No basis given
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Overall, almost half of the participants were not able to provide the correct interpretation of what the model did using the SMC alone.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“with participants echoing the concerns that were previously articulated by the experts in Section 3.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Supporting interpretability is complex but nearly all (n=7) participants indicated that examples could provide some useful guidance.”
hedgedn = 7- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“A few participants (n=4) expressed that text, visuals, and interaction options led to information overload.”
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“The reference model card (Section 3.1), here acting as the SMC, and the IMC each had features that were preferred by both types of analysts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Compared to the prior study, non-expert analysts provided greater depth towards the themes of stakeholders ; design considerations ; and sensemaking & skepticism in particular.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“participants indicated that interactivity added substantially to the SCM because it allowed participants to 'play with the data and see the results [in a way that] that feels super intuitive instead of explaining everything' [A1-09].”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One of the most salient implications of the IMC design was that it helped participants accurately understand what the model was and what it did.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants indicated that various stakeholders would need help making sense of the information in the model card.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Even three participants who did correctly describe the model were not confident in their interpretation”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- It may state two separate claims as one. 24
- Probably not a substantial claim. 35
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Notably, the addition of guiding elements, suggested in the previous study (Table 2), were well received and did help participants understand and interpret the model card information.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our conceptual study emphasized the importance of information design within both standard and interactive model cards.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants indicated that for these issues and overall, greater attention to information, visual, and interaction design would improve trust and safety.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Several (n=4) participants expressed that it could be beneficial to add sensitive populations to the disaggregated model performance, but they suggested caution since the definition of these groups was culturally and geographically contextual.”
hedgedn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants suggested that integrating the model's code and files, present in the SMC, would be valuable for the IMC.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Warnings, prompts, and summaries are all examples of guidance that participants suggested could help guide analysts in making sense of model cards.”
hedged- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“When asked, nearly all participants (n=15) indicated that the IMC was something they would use in their daily work, while some (n=2) indicated it was context-dependent.”
hedgedn = 15- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“one participant initially thought the model performed well for a particular subpopulation, then saw the small sample size and revised their interpretation.”
plaindirectSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“the interactivity would enable people to 'elicit more questions' [E04] that allowed them to form and test their mental models about the data and better 'see the things that model lacks' [E05].”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Participants articulated that several components of the IMC helped them engage with the model and data more effectively by clarifying and prioritizing information, particularly: layout of information (n=13); language (n=6); ability to add examples (n=15); and visual and interactive elements (n=16).”
plaindirectn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“experts saw the opportunity to use interaction as a way to help non-expert analysts develop their conceptual understanding of the NLP model.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Three participants voiced the importance of being able to contest the model's results, and wanted to see this ability expanded beyond adding sentences (Section 4.1) to the model's training and testing data or uploading their own dataset.”
plainn = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Increasingly, such statements are seen as a 'cop-out because it just kind of says 'there is some bias somewhere' which is not entirely helpful.' [E06]”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Nearly all (n=16) participants also made suggestions for improvements to the IMC design. These included changes to elements of the interface that were confusing (n=5), the layout (n=6), and types of guidance (n=1) provided.”
plainn = 16- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“they emphasized the need to prominently display the total size of these subpopulations (and others) to support valid interpretation.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
The paper states this without a source or data behind it here, so there is nothing to check it against. The quote shows it is in the paper.
In the paper
“We found that our approach to guidance, shareability, and traceability begins to address these needs, and we surfaced additional relevant implications for design.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
- Rests on an assumption: The DistilBERT SST-2 sentiment analysis task is representative of NLP model development regimens for other tasks.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants expressed doubt that a non-expert analyst would understand the model card or know how to act on the information it contained.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, a sense of focus/overwhelm varied among participants, and one participant articulated that the IMC contained 'too much information and is very cluttered' [E04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Two participants suggested stating explicitly whether bias assessments had been conducted, and one suggested raising a warning if such bias analyses are not run.”
plainn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Four participants stated that a summary of the model's purpose (i.e., 'this is a sentiment analysis model') was needed and that it would improve the readability of the model card.”
plaindirectn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, having an effective default example to prompt end-users was also important [E04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Although the IMC included templates to examine ethically challenging examples (Section 4.1), only four participants expressed that these features could stimulate discussion.”
plainn = 4- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“While the IMC also provided template sentences to promote ethical thinking, participants largely favored their own examples.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Moreover, three participants expressed that IMCs, if they could be easy to produce, would lower their existing model documentation burden.”
hedgedconditionaln = 3- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, our research supports prior findings that these concepts are challenging to integrate into decision-making [48, 60], and we identified some cynicism toward ethics statements, as well as assertions that ethical questions require contextual, situated examination.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Others felt that topics of safety and ethics were too abstract, with one participant stating that they 'wouldn't worry too much about that to be honest, I guess it depends on the type of analysis you're doing.' [A2-05].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Here, the reference model card contained information and actions to obtain the model code and launch it - something that was absent in the IMC.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Unexpectedly, two participants pushed back against the inclusion of more actionable ethical and safety content, articulating concerns that too much guidance could be detrimental and provide a false sense of security.”
plaindirectn = 2- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“One participant expressed doubt that any model documentation could really give an adequate picture of ethical and safety risks, emphasizing that ' 'this [model card] would not be my only data point or else, you know, that's a very quick way to lose your job' [A1-04].”
plainSingle case- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, to be sure that this content is effectively understood by a wide audience, the hierarchy and design of this content is equally important.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Video transcripts were automatically generated, including speaker identification, and were verified for accuracy by the authors.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“This depth allowed us to expand on the subthemes from Table 2 and broaden our conceptions of how model cards might be used.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“All participants had exposure to documentation for ML/AI models in some form, but only six were familiar with model cards and two with RGRs.”
plainn = 10- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
In the paper
“Robustness Gym helps model developers report on model behavior through the generation of robustness reports (RGRs) that summarize model performance on various data slices [23].”
What the sources say
- Not checked yetcites [23]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Clear guidance can support non-experts in understanding and knowing how to act on the information.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Additional guidance will be necessary to support interpretability of the outputs.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: SMCs have been primarily tailored toward developers and machine learning experts.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that this productive skepticism would make ideas of intended and unintended uses more intuitive than simply being told what they are.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper may give no reason why it holds.
- Rests on an assumption: Not all readers of a model card will want to interact with it.
None of the sources cited for this claim have been checked yet.
In the paper
“Prior work has shown that non-expert analysts benefit from interacting with machine learning models and their data [1, 19, 59].”
What the sources say
- Not checked yetcites [1]
- Not checked yetcites [19] source
- Not checked yetcites [59]
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim.
None of the sources cited for this claim have been checked yet.
In the paper
“As many researchers have articulated, safe and effective deployment of AI/ML systems in practice requires a robust understanding of different stakeholders' needs and goals for explainability [8, 32]; support for traceability and auditing [71]; and strategies to address concerns within existing organizational structures and dynamics [8, 31, 32, 57, 71].”
What the sources say
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, we found that scaffolding these interactions to support sensemaking requires clear and actionable guidance.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“1) Interactive model cards are a bridge to further analysis, not a substitute.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We also recognize that creating and learning how to use IMCs is a heavier burden than for text-based model cards.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: The recruited participants are representative of the future data workforce.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For model cards to be adopted and impact organizational processes, they must support knowledge sharing and negotiation across stakeholders with diverse backgrounds and perspectives.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We argue that with thoughtfully designed interaction, model cards can be a first step for data workers to explore how trustworthy a model is and whether it might meet their needs, serving as a guide to deeper interrogation when desired.”
plainconditional- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Our findings indicate, unsurprisingly, that model documentation can spark ethical thinking, but will never substitute for it.”
hedgeddirect- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
What the paper did — it cites no source for this, so there is none to check. The quote shows it is in the paper.
In the paper
“Design choices and the choice of sentiment analysis tasks (Section 3.1) likely influenced our findings.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“For individual stakeholders, the most effective form for interactivity in model cards will be situated and contextual.”
plaindirect- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: fails — The paper may give no reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence. The paper’s conclusions may not depend on it. The paper may give no reason why it holds.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Importantly, our analysis reinforces that considerations of ethics are personal and contextual, so they are unlikely to be addressed with a single approach.”
hedgedconditional- Asserted: passes — The authors put it forward as true.
- Checkable: unclear — It is unclear whether it could be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
It may state two separate claims as one.
- Rests on an assumption: Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Their feedback validates many of our design choices, but also offers avenues for further improvements.”
plaincorrelational- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: fails — The paper’s conclusions may not depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. The paper’s conclusions may not depend on it. It may state two separate claims as one.
A conclusion the paper draws from its own results — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“However, we believe that the broader design guidelines (Section 3.4) and conceptual dimensions that we surfaced (Tables 2, 4, and 5) are robust and will generalize to other tasks and contexts.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: fails — It may be too vague to check against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
Probably not a substantial claim. It may be too vague to check against evidence.
- Rests on an assumption: The recruited participants are representative of the future data workforce.
The paper relies on this without support.
Model cards and similar documentation rely on the developer to accurately and clearly report on the model and its performance.
“However, these model cards and similar forms of documentation rely on the developer to accurately and clearly report on the model and its performance.”
The paper relies on this without support.
Many important details, such as unintended uses or disaggregated model performance, are omitted from model cards.
“Often, this process is labor-intensive and many important details, such as unintended uses or disaggregated model performance, are omitted.”
The paper relies on this without support.
The DistilBERT SST-2 sentiment analysis task is representative of NLP model development regimens for other tasks.
“We chose to focus on a sentiment analysis task because it is widely known and easily understood, while still being representative of NLP model development regimens for other tasks.”
The paper relies on this without support.
The design probes (SMC, RGR, IMC) adequately represent the respective documentation types.
“We developed design probes [70] of an SMC, RGR, and IMC. For the SMC, we used the all the text from the reference model card, with some exceptions, laid out according to the specifications in Mitchell et al. [44].”
The paper relies on this without support.
The model card format is text-based and concise, making it both broadly accessible and applicable across model types.
“The format they propose is text-based and concise, making it both broadly accessible and applicable across model types.”
The paper relies on this without support.
Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.
“Prior work related to SMCs has focused primarily on the categories and depth of information that model documentation should include.”
The paper relies on this without support.
SMCs have been primarily tailored toward developers and machine learning experts.
“SMCs have been primarily tailored toward developers and machine learning experts”
The paper relies on this without support.
Not all readers of a model card will want to interact with it.
“Not all readers of a model card will want to interact with it.”
The paper relies on this without support.
Technology that enables low or no code data analysis is lowering barriers to developing and deploying deep learning models.
“Technology that enables low or no code data analysis is lowering barriers to developing and deploying deep learning models.”
The paper relies on this without support.
The data workers taking on AI/ML development do not have formal education in the theoretical and technical underpinnings of these systems.
“Increasingly, the data workers taking on AI/ML development do not have formal education in the theoretical and technical underpinnings of these systems.”
The paper relies on this without support.
The recruited participants are representative of the future data workforce.
“We recruited and explored the perspectives of people who we believe are representative of the future data workforce.”
The paper relies on this without support.
Model cards are intended to surface ethical and safety implications, a topic especially pressing for deep learning models.
“Model cards are intended to surface, amongother things, ethical and safety implications - a topic that is especially pressing for deep learning models.”
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source says what the claim attributes to it.
In the citing paper
“Model cards represent the field's standard for disclosing and documenting key facts about the architecture, training and evaluation of the model [14, 40].”
From Rethinking open source generative AI: open washing and the EU AI Act (2024) · cites this paper as [14]
Citations
Cites
Harris, C. R., Millman, K. J., van der Walt, S. J., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N. J., et al. (2020) · Nature · doi:10.1038/s41586-020-2649-2 · Cited by 23,826
Harris, C., Millman, K., van der Walt, S., Gommers, R., Virtanen, P., Cournapeau, D., Wieser, E., Taylor, J., Berg, S., Smith, N., et al. (2020) · RePEc: Research Papers in Economics · Cited by 17,844
Data Structures for Statistical Computing in Python
McKinney, W. (2010) · Proceedings of the Python in Science Conferences · doi:10.25080/majora-92bf1922-00a · Cited by 12,930
Barredo Arrieta, A., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., et al. (2020) · Information Fusion · doi:10.1016/j.inffus.2019.12.012 · Cited by 10,423
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741
Scikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜
Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445922 · Cited by 6,988
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., & Potts, C. (2013) · Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d13-1170 · Cited by 6,878
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter
Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019) · arXiv · doi:10.48550/arxiv.1910.01108 · Cited by 4,653
Barocas, S., & Selbst, A. D. (2014) · doi:10.2139/ssrn.2477899 · Cited by 3,048
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 9×
“They were then shown examples of a standard model card, a Robustness Gym report, and our concept for an interactive model card (Section 3.1).”
BackgroundInfluentialOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daumé, H., & Crawford, K. (2018) · arXiv · doi:10.48550/arxiv.1803.09010 · Cited by 2,474
Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., III, H. D., & Crawford, K. (2021) · Communications of the ACM · doi:10.1145/3458723 · Cited by 2,469
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
The CARE Principles for Indigenous Data Governance
Carroll, S. R., Garba, I., Figueroa-Rodríguez, O. L., Holbrook, J., Lovett, R., Materechera, S., Parsons, M., Raseroka, K., Rodriguez-Lonebear, D., Rowe, R., et al. (2020) · Data Science Journal · doi:10.5334/dsj-2020-043 · Cited by 1,408
Zhang, Y., Liao, Q. V., & Bellamy, R. K. E. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372852 · Cited by 895
Mittelstadt, B., Russell, C., & Wachter, S. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287574 · Cited by 794
“Everyone wants to do the model work, not the data work”: Data Cascades in High-Stakes AI
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445518 · Cited by 781
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777
Shneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Manipulating and Measuring Model Interpretability
Poursabzi-Sangdeh, F., Goldstein, D. G., Hofman, J. M., Wortman Vaughan, J. W., & Wallach, H. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445315 · Cited by 728
Raji, I. D., & Buolamwini, J. (2019) · Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society · doi:10.1145/3306618.3314244 · Cited by 651
Explainable machine learning in deployment
Bhatt, U., Xiang, A., Sharma, S., Weller, A., Taly, A., Jia, Y., Ghosh, J., Puri, R., Moura, J. M. F., & Eckersley, P. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3375624 · Cited by 633
Kaur, H., Nori, H., Jenkins, S., Caruana, R., Wallach, H., & Wortman Vaughan, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376219 · Cited by 564
Formalizing Trust in Artificial Intelligence: Prerequisites, Causes and Goals of Human Trust in AI
Jacovi, A., Marasović, A., Miller, T., & Goldberg, Y. (2021) · Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency · doi:10.1145/3442188.3445923 · Cited by 545
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse
Hoffmann, A. L. (2019) · Information, Communication & Society · doi:10.1080/1369118x.2019.1573912 · Cited by 535 · cited 1×
“The potential for both broad and deep harm that AI/ML systems can pose to vulnerable people and ecosystems, especially when impacted people do not have the opportunity to interrogate and contest decisions driven by AI/ML(e.g., [4, 25, 26, 29, 72]).”
BackgroundProceedings of the 2020 Conference on Fairness, Accountability, and Transparency
Mireille Hildebrandt, C. C., & del Castillo, C. C. (2020) · doi:10.1145/3351095 · Cited by 486
Social Justice-Oriented Interaction Design
Dombrowski, L. S., Harmon, E., & Fox, S. (2016) · Designing Interactive Systems Conference · doi:10.1145/2901790.2901861 · Cited by 475
The Measure and Mismeasure of Fairness
Corbett-Davies, S., Gaebler, J. D., Nilforoshan, H., Shroff, R., & Goel, S. (2018) · arXiv · doi:10.48550/arxiv.1808.00023 · Cited by 444
Assessing risk, automating racism
Benjamin, R. (2019) · Science · doi:10.1126/science.aaz3873 · Cited by 378
A Review of User Interface Design for Interactive Machine Learning
Dudley, J. J., & Kristensson, P. O. (2018) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3185517 · Cited by 343
Rakova, B., Yang, J., Cramer, H., & Chowdhury, R. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449081 · Cited by 329
Towards a critical race methodology in algorithmic fairness
Hanna, A., Denton, R., Smart, A., & Smith-Loud, J. (2020) · Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency · doi:10.1145/3351095.3372826 · Cited by 295
Sokol, K., & Flach, P. A. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372870 · Cited by 278
Ethics-Based Auditing to Develop Trustworthy AI
Mökander, J., & Floridi, L. (2021) · Minds and Machines · doi:10.1007/s11023-021-09557-8 · Cited by 268
Hate speech detection and racial bias mitigation in social media based on BERT model
Mozafari, M., Farahbakhsh, R., & Crespi, N. (2020) · PLOS ONE · doi:10.1371/journal.pone.0237861 · Cited by 260
Wallace, J., McCarthy, J., Wright, P., & Olivier, P. L. (2013) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2470654.2466473 · Cited by 245
Gender Recognition or Gender Reductionism?
Hamidi, F., Scheuerman, M. K., & Branham, S. M. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173582 · Cited by 240
Altair: Interactive Statistical Visualizations for Python
Vanderplas, J., Granger, B., Heer, J., Moritz, D., Wongsuphasawat, K., Satyanarayan, A., Lees, E., Timofeev, I., Welsh, B., & Sievert, S. (2018) · The Journal of Open Source Software · doi:10.21105/joss.01057 · Cited by 220
Human factors in model interpretability: Industry practices, challenges, and needs
Hong, S. R., Hullman, J., & Bertini, E. (2020) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3392878 · Cited by 219
Shen, H., DeVos, A., Eslami, M., & Holstein, K. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3479577 · Cited by 171
What you see is what you can change: Human-centered machine learning by interactive visualization
Sacha, D., Sedlmair, M., Zhang, L., Lee, J. A., Peltonen, J., Weiskopf, D., North, S. C., & Keim, D. A. (2017) · Neurocomputing · doi:10.1016/j.neucom.2017.01.105 · Cited by 164
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156
Knijnenburg, B. P., Reijmer, N. J. M., & Willemsen, M. C. (2011) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/2043932.2043960 · Cited by 132
Morley, J., Floridi, L., Kinsey, L., & Elhalal, A. (2019) · SSRN Electronic Journal · doi:10.2139/ssrn.3830348 · Cited by 120
Nourani, M., King, J., & Ragan, E. (2020) · Proceedings of the AAAI Conference on Human Computation and Crowdsourcing · doi:10.1609/hcomp.v8i1.7469 · Cited by 116
Racial categories in machine learning
Benthall, S., & Haynes, B. D. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287575 · Cited by 110
Designing Contestability: Interaction Design, Machine Learning, and Mental Health
Hirsch, T., Merced, K., Narayanan, S., Imel, Z. E., & Atkins, D. C. (2017) · Proceedings of the 2017 Conference on Designing Interactive Systems · doi:10.1145/3064663.3064703 · Cited by 110
Anchoring Bias Affects Mental Model Formation and User Reliance in Explainable AI Systems
Nourani, M., Roy, C., Block, J. E., Honeycutt, D. R., Rahman, T., Ragan, E., & Gogate, V. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450639 · Cited by 107 · cited 2×
“The opportunity to “be really curious to just kind of play around” [A2-05] can help mitigate anchoring bias [52].”
BackgroundAn Action-Oriented AI Policy Toolkit for Technology Audits by Community Advocates and Activists
Krafft, P. M., Young, M., Katell, M. A., Lee, J. E., Narayan, S., Epstein, M., Dailey, D., Herman, B., Tam, A., Guetler, V., et al. (2021) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3442188.3445938 · Cited by 101
Czarnowska, P., Vyas, Y., & Shah, K. (2021) · Transactions of the Association for Computational Linguistics · doi:10.1162/tacl_a_00425 · Cited by 72
Explainability pitfalls: beyond dark patterns in explainable AI
Ehsan, U., & Riedl, M. O. (2024) · Patterns · doi:10.1016/j.patter.2024.100971 · Cited by 72
Fits and Starts: Enterprise Use of AutoML and the Role of Humans in the Loop
Crisan, A., & Fiore-Gartland, B. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445775 · Cited by 56
Self-citationDynaSent: A Dynamic Benchmark for Sentiment Analysis
Potts, C., Wu, Z., Geiger, A., & Kiela, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.186 · Cited by 56
Lučić, A., Haned, H., & de Rijke, M. (2020) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3351095.3372824 · Cited by 46
Communicating with Interactive Articles
Hohman, F., Conlen, M., Heer, J., & Chau, D. H. (2020) · Distill · doi:10.23915/distill.00028 · Cited by 46
The Problem with Metrics is a Fundamental Problem for AIPreprint
Thomas, R., & Uminsky, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2002.08512 · Cited by 37
Designing Interactive Transfer Learning Tools for ML Non-Experts
Mishra, S., & Rzeszotarski, J. M. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445096 · Cited by 34
Wang, J., & Moulden, A. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3443452 · Cited by 18
Auditing Digital Platforms for Discrimination in Economic Opportunity AdvertisingPreprint
Kingsley, S., Wang, C., Mikhalenko, A., Sinha, P., & Kulkarni, C. E. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2008.09656 · Cited by 16
How do People Train a Machine?
Sanchez, T., Caramiaux, B., Françoise, J., Bevilacqua, F. P., & Mackay, W. E. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449236 · Cited by 16
Certified Robustness to Programmable Transformations in LSTMs
Zhang, Y., Albarghouthi, A., & D’Antoni, L. (2021) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2021.emnlp-main.82 · Cited by 8
Conceptualising Contestability: Perspectives on Contesting Algorithmic Decisions
Lyons, H., Velloso, E., & Miller, T. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449180 · Cited by 4
Research works printed but not linked: 14
- Saleema Amershi, Maya Cakmak, W. Bradley Knox, and Todd Kulesza. 2014. Power to the People: The Role of Humans in Interactive Machine Learning. AI Mag. 35(2014), 105–120.
- Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. ” O’Reilly Media, Inc.”.
- Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey. 2015. Algorithm aversion: people erroneously avoid algorithms after seeing them err.Journal of experimental psychology. General 144 1 (2015), 114–26.
- Alexandra Giannopoulou. 2020. Algorithmic Systems: the Consent is in the Detail?Internet Policy Review 9, 1 (2020).
- Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021. Robustness Gym: Unifying the NLP Evaluation Landscape. arXiv preprint arXiv:2101.04840(2021). · arXiv
- Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. 2018. A Survey of Methods for Explaining Black Box Models. ACM computing surveys (CSUR) 51, 5 (2018), 1–42.
- Min Kyung Lee, Daniel Kusbit, Anson Kahng, Ji Tae Kim, Xinran Yuan, Allissa Chan, Daniel See, Ritesh Noothigattu, Siheon Lee, Alexandros Psomas, 2019. WeBuildAI: Participatory Framework for Algorithmic Governance. Proceedings of the ACM on Human-Computer Interaction 3, CSCW(2019), 1–35.
- Andreas Madsen Siva Reddy and Sarath Chandar. 2021. Post-hoc Interpretability for Neural NLP: A Survey. arXiv preprint arXiv:2108.04840(2021). · arXiv
- Christoph Molnar, Giuseppe Casalicchio, and Bernd Bischl. 2020. Interpretable Machine Learning–A Brief History, State-of-the-Art and Challenges. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 417–431.
- Michael Muller. 2014. Curiosity, Creativity, and Surprise as Analytic Tools: Grounded Theory Method. In Ways of Knowing in HCI. Springer, 25–48.
- Radim Rehurek and Petr Sojka. 2011. Gensim–python framework for vector space modelling. NLP Centre, Faculty of Informatics, Masaryk University, Brno, Czech Republic 3, 2(2011).
- Salome Viljoen. 2021. A Relational Theory of Data Governance. The Yale Law Journal 131, 573 (2021).
- Meredith Whittaker, Kate Crawford, Roel Dobbe, Genevieve Fried, Elizabeth Kaziunas, Varoon Mathur, Sarah Mysers West, Rashida Richardson, Jason Schultz, and Oscar Schwartz. 2018. AI Now Report 2018. AI Now Institute at New York University New York.
- Jilei Yang, Diana Negoescu, and Parvez Ahammad. 2021. Intellige: A User-Facing Model Explainer for Narrative Explanations. arXiv preprint arXiv:2105.12941(2021). · arXiv
Cited by
Sorted by citations
Towards conversational diagnostic artificial intelligence
Tu, T., Schaekermann, M., Palepu, A., Saab, K., Freyberg, J., Tanno, R., Wang, A., Li, B., Amin, M., Cheng, Y., et al. (2025) · Nature · doi:10.1038/s41586-025-08866-7 · Cited by 366
What is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“Transparency can be achieved through data or model cards, documentation of the data and who is represented within it, as well as how the model was made, assumptions that were made, how the model behaves in relation to diverse groups of people, and how well the model performs in relation to those groups [45, 55, 68, 10…”
BackgroundAuditing large language models: a three-layered approach
Mökander, J., Schuett, J., Kirk, H. R., & Floridi, L. (2024) · AI and Ethics · doi:10.1007/s43681-023-00289-2 · Cited by 289
“, interactive model cards [183] and information about the initial training dataset [184], [185], to help downstream developers adapt the model appropriately.”
BackgroundUnderstanding accountability in algorithmic supply chains
Cobbe, J., Veale, M., & Singh, J. P. (2023) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3593013.3594073 · Cited by 141
“or using a system, such as for datasheets [39] or data cards [86] to describe datasets, or model cards [29, 75] and factsheets [7] to describe model specification and capabilities.”
BackgroundMethodLiesenfeld, A., Lopez, A., & Dingemanse, M. (2023) · Proceedings of the 5th International Conference on Conversational User Interfaces · doi:10.1145/3571884.3604316 · Cited by 111
“Interactive model cards [13] offer a promising step towards a humancentered approach to documentation.”
Background
Metadata
- Authors
- Crisan, A., Drouhard, M., Vig, J., & Rajani, N.
- Affiliations
- Tableau Software (United States) (US)
- Salesforce (United States) (US)
- Venue
- 2022 ACM Conference on Fairness Accountability and Transparency
- Publisher
- ACM
- Published
- 2022-06-20
- Pages
- 427-439
- OpenAlex
- W4229447062
- Cited by
- 101
- References
- 56
- Citations by year
- 2026: 24 · 2025: 31 · 2024: 22 · 2023: 23 · 2022: 1
- FWCI
- 10.64
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM Conference on Fairness, Accountability, and Transparency (FAccT) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Physical Sciences › Computer Science › Artificial Intelligence › Ethics and Social Impacts of AI
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Keywords
- model documentation, human-centered design, non-expert users, deep learning models, natural language processing, AI ethics, model interpretability, trust and safety, design guidelines, stakeholder perspectives, organizational dynamics, socio-technical context
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
A language model drew the claims from the paper on 2026-10-06, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-06 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
