Planning for Natural Language Failures with the AI Playbook
Abstract
Prototyping AI user experiences is challenging due in part to probabilistic AI models making it difficult to anticipate, test, and mitigate AI failures before deployment. In this work, we set out to support practitioners with early AI prototyping, with a focus on natural language (NL)-based technologies. Our interviews with 12 NL practitioners from a large technology company revealed that, in addition to challenges prototyping AI, prototyping was often not happening at all or focused only on idealized scenarios due to a lack of tools and tight timelines. These findings informed our design of the AI Playbook, an interactive and low-cost tool we developed to encourage proactive and systematic consideration of AI errors before deployment. Our evaluation of the AI Playbook demonstrates its potential to 1) encourage product teams to prioritize both ideal and failure scenarios, 2) standardize the articulation of AI failures from a user experience perspective, and 3) act as a boundary object between user experience designers, data scientists, and engineers.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
While research has started to categorize types of breakdowns in communication between users and NLP agents, little work has looked into how users perceive these failures and subsequently trust and use their voice assistants.
“Further, while research has started to categorize types of breakdowns in communication between users and NLP agents [28, 46], little work has looked into how users perceive these failures and subsequently trust and use their voice assistants.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [28]
The dataset specifically addresses failures that occur with voice assistants.
“While prior work [28] was useful in helping NLP practitioners anticipate and plan for failures across many types of NLP technologies, our dataset specifically addresses failures that occur with voice assistants.”
From A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures (2023) · cites this paper as [28]
Cites
Using thematic analysis in psychology
Braun, V., & Clarke, V. (2006) · Qualitative Research in Psychology · doi:10.1191/1478088706qp063oa · Cited by 194,414 · see citing works
Model Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1× · see citing works
“Despite the importance of such errors, a full understanding of this space is an active area of research [14, 19, 21], and we welcome expanding the AI Playbook to better predict in which circumstances such errors may arise, and to provide better guidance on how they can be simulated and ultimately…”
BackgroundGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · see citing works
Self-citationPrinciples of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490 · see citing works
(2007) · International Linear Collider · Cited by 1,382 · see citing works
Gestures without libraries, toolkits or training: a $1 recognizer for user interface prototypes
Wobbrock, J. O., Wilson, A. D., & Li, Y. (2007) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/1294211.1294238 · Cited by 823 · see citing works
Wizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines
Riek, L. D. (2012) · Journal of Human-Robot Interaction · doi:10.5898/jhri.1.1.riek · Cited by 778 · see citing works
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need?
Holstein, K., Wortman Vaughan, J., Daumé, H., Dudik, M., & Wallach, H. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300830 · Cited by 777 · see citing works
An iterative design methodology for user-friendly natural language office information applications
Kelley, J. F. (1984) · ACM Transactions on Information Systems · doi:10.1145/357417.357420 · Cited by 713 · see citing works
Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376301 · Cited by 622 · see citing works
Fraser, N., & Gilbert, N. (1991) · Computer Speech & Language · doi:10.1016/0885-2308(91)90019-m · Cited by 440 · see citing works
Madaio, M. A., Stark, L., Wortman Vaughan, J., & Wallach, H. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376445 · Cited by 436 · see citing works
UX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · see citing works
Performance issues and error analysis in an open-domain question answering system
MOLDOVAN, D. I., PAŞCA, M. A., Harabagiu, S. M., & Surdeanu, M. (2003) · ACM Transactions on Information Systems · doi:10.1145/763693.763694 · Cited by 233 · see citing works
Investigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. M. (2018) · Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 230 · see citing works
Composing letters with a simulated listening typewriter
Gould, J. D., Conti, J., & Hovanyecz, T. (1983) · Communications of the ACM · doi:10.1145/2163.358100 · Cited by 225 · see citing works
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · see citing works
Automatic Speech Recognition Errors Detection and Correction: A Review
Errattahi, R., El Hannani, A., & Ouahmane, H. (2018) · Procedia Computer Science · doi:10.1016/j.procs.2018.03.005 · Cited by 155 · cited 3× · see citing works
“Finally, we generated guidance for simulating failures, building on techniques suggested in previous work (e.g., [2, 8, 13, 22]).”
BackgroundMethodConversation as Action Under Uncertainty
Paek, T., & Horvitz, E. J. (2013) · arXiv · doi:10.48550/arxiv.1301.3883 · Cited by 145 · see citing works
Sorry, I Didn’t Catch That! An Investigation of Non-understanding Errors and Recovery Strategies
Bohus, D., & Rudnicky, A. I. (2018) · doi:10.1184/r1/6609683 · Cited by 109 · see citing works
Calendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., & Monroy-Hernández, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025780 · Cited by 106 · see citing works
Sketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 2× · see citing works
“As previously described, however, prior work has highlighted numerous challenges in prototyping AI interactions due in part to difculties knowing what behaviors and error conditions might manifest and should be sketched out [28].”
BackgroundSelf-citationA design tool for camera-based interaction
Fails, J. A., & Olsen, D. R. (2003) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/642611.642690 · Cited by 80 · see citing works
Lasecki, W. S., Kim, J., Rafter, N., Sen, O., Bigham, J. P., & Bernstein, M. S. (2015) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2702123.2702565 · Cited by 78 · see citing works
Towards Taxonomy of Errors in Chat-oriented Dialogue Systems
Higashinaka, R., Funakoshi, K., Araki, M., Tsukahara, H., Kobayashi, Y., & Mizukami, M. (2015) · Annual Meeting of the Special Interest Group on Discourse and Dialogue · doi:10.18653/v1/w15-4611 · Cited by 67 · see citing works
Calendar.Help: Designing a Workflow-Based Scheduling Agent with Humans in the LoopPreprint
Cranshaw, J., Kocielnik, R., Yu, B., & Teevan, J. B. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940295 · Cited by 63 · see citing works
Wizard of Oz Experimentation for Language Technology Applications: Challenges and Tools
Schlögl, S., Doherty, G. J., & Luz, S. F. (2014) · Interacting with Computers · doi:10.1093/iwc/iwu016 · Cited by 43 · see citing works
Zemčík, T. (2020) · AI & Society · doi:10.1007/s00146-020-01053-4 · Cited by 36 · see citing works
Toward a Taxonomy of Communication Errors
Paek, T. (2003) · Cited by 22 · see citing works
Flagged first, then by citations
1 of the 30 references couldn’t be matched to a known paper.
Cited by
Sorted by citations
PromptMaker: Prompt-based Prototyping with Large Language Models
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503564 · Cited by 157 · see citing works
Requirements engineering for artificial intelligence systems: A systematic mapping study
Ahmad, K., Abdelrazek, M., Arora, C., Bano, M., & Grundy, J. (2023) · Information and Software Technology · doi:10.1016/j.infsof.2023.107176 · Cited by 132 · see citing works
Investigating How Practitioners Use Human-AI Guidelines: A Case Study on the People + AI Guidebook
Yildirim, N., Pushkarna, M., Goyal, N., WATTENBERG, M. P., & Viégas, F. B. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580900 · Cited by 107 · see citing works
Designing Responsible AI: Adaptations of UX Practice to Meet Responsible AI Challenges
Wang, Q., Madaio, M., Kane, S. K., Kapania, S., Terry, M. A., & Wilcox, L. G. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581278 · Cited by 94 · see citing works
Liao, Q. V., Subramonyam, H., Wang, J., & Vaughan, J. W. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580652 · Cited by 97 · see citing works
Creating Design Resources to Scaffold the Ideation of AI Concepts
Yildirim, N., Oh, C., Sayar, D., Brand, K., Challa, S., Turri, V., Walton, N., Wong, A. E., Forlizzi, J., McCann, J., et al. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596058 · Cited by 77 · see citing works
How Experienced Designers of Enterprise Applications Engage AI as a Design Material
Yildirim, N., Kass, A., Tung, T., Upton, C., Costello, D., Giusti, R., Lacin, S., Lovic, S., O'Neill, J. M., Meehan, R. O., et al. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517491 · Cited by 71 · see citing works
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Wang, Z. J., Kulkarni, C., Wilcox, L., Terry, M., & Madaio, M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642335 · Cited by 55 · see citing works
Li, T., Vorvoreanu, M., DeBellis, D., & Amershi, S. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3511605 · Cited by 55 · see citing works
Self-citationTowards a multi-stakeholder value-based assessment framework for algorithmic systems
Yurrita, M., Murray-Rust, D., Balayn, A., & Bozzon, A. (2022) · ACM Conference on Fairness, Accountability, and Transparency (FAccT) · doi:10.1145/3531146.3533118 · Cited by 42 · see citing works
What Did My AI Learn? How Data Scientists Make Sense of Model Behavior
Cabrera, Á. A., Ribeiro, M. T., Lee, B., DeLine, R. A., Perer, A., & Drucker, S. M. (2022) · ACM Transactions on Computer-Human Interaction · doi:10.1145/3542921 · Cited by 41 · see citing works
Feng, K. J. K., Coppock, M. J., & McDonald, D. W. (2023) · Designing Interactive Systems Conference · doi:10.1145/3563657.3596101 · Cited by 41 · see citing works
Seamful XAI: Operationalizing Seamful Design in Explainable AI
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637396 · Cited by 61 · see citing works
PromptInfuser: Bringing User Interface Mock-ups to Life with Large Language Models
Petridis, S., Terry, M. A., & Cai, C. J. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585628 · Cited by 40 · see citing works
Berman, G., Goyal, N., & Madaio, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642398 · Cited by 45 · see citing works
A study of UX practitioners roles in designing real-world, enterprise ML systems
Zdanowska, S., & Taylor, A. S. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3517607 · Cited by 39 · see citing works
Moore, S., Liao, Q. V., & Subramonyam, H. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581242 · Cited by 40 · see citing works
PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers’ Workflows
Petridis, S., Terry, M. A., & Cai, C. J. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661613 · Cited by 40 · see citing works
A Mixed-Methods Approach to Understanding User Trust after Voice Assistant Failures
Baughan, A., Wang, X., Liu, A., Mercurio, A., Chen, J., & Ma, X. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581152 · Cited by 32 · see citing works
“We determined they were not sufficient for our purposes, so we proceeded to crowdsource a dataset of failures, adapting a taxonomy from Hong et al. [28] to guide our collection.”
BackgroundMethodInfluential(2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642476 · Cited by 30 · see citing works
ASHABot: An LLM-Powered Chatbot to Support the Informational Needs of Community Health Workers
Ramjee, P., Chhokar, M., Sachdeva, B., Meena, M., Abdullah, H., Vashistha, A., Nagar, R., & Jain, M. (2025) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713680 · Cited by 35 · see citing works
Feffer, M., Martelaro, N., & Heidari, H. (2023) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3617694.3623223 · Cited by 25 · see citing works
Tankelevitch, L., Glassman, E. L., He, J., Kazemitabaar, M., Kittur, A., Lee, M., Palani, S., Sarkar, A., Ramos, G., Rogers, Y., et al. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706745 · Cited by 24 · see citing works
Nishal, S., Sinchai, J., & Diakopoulos, N. A. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3637419 · Cited by 22 · see citing works
Zaimah, N. R., Hartanto, E. B., & Zahro, F. (2023) · Mantiqu Tayr Journal of Arabic Language · doi:10.25217/mantiqutayr.v4i1.3951 · Cited by 13 · see citing works
Madaio, M., Chen, J., Wallach, H., & Vaughan, J. W. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3653705 · Cited by 13 · see citing works
Conversations with the News: Co-speculation into Conversational Interactions with News Content
Nordberg, O. E., & Guribye, F. (2023) · ACM Conference on Conversational User Interfaces (CUI) · doi:10.1145/3571884.3597123 · Cited by 10 · see citing works
AI-Driven Support for People with Speech & Language Difficulties
Dangol, A., Huang, Y., Setlur, S., Smolansky, A., Subramonyam, H., Suh, H., Xiong, J., & Kientz, J. A. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3643984 · Cited by 10 · see citing works
Truss, M., & Schmitt, M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2425454 · Cited by 11 · see citing works
Requirements Elicitation and Modelling of Artificial Intelligence Systems: An Empirical Study
Ahmad, K., Abdelrazek, M., Arora, C., Grundy, J., & Bano, M. (2023) · International Conference on Evaluation of Novel Approaches to Software Engineering · doi:10.5220/0011842300003464 · Cited by 10 · see citing works
Deng, W. H., Barocas, S., & Vaughan, J. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711076 · Cited by 12 · see citing works
Weekes, T. R., & Eskridge, T. C. (2022) · Lecture notes in computer science · doi:10.1007/978-3-031-21707-4_41 · Cited by 10 · see citing works
Seamful XAI: Operationalizing Seamful Design in Explainable AIPreprint
Ehsan, U., Liao, Q. V., Passi, S., Riedl, M. O., & Daumé, H. (2022) · arXiv (Cornell University) · doi:10.13016/at4q-63xb · Cited by 8 · see citing works
Analyzing Collaborative Challenges and Needs of UX Practitioners when Designing with AI/ML
Muralikumar, M. D., & McDonald, D. W. (2024) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3686986 · Cited by 7 · see citing works
An Emerging Design Space of How Tools Support Collaborations in AI Design and Development
Muralikumar, M. D., & McDonald, D. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701181 · Cited by 7 · see citing works
Impact Assessment Card: Communicating Risks and Benefits of AI Uses
Bogucka, E. P., Constantinides, M., Šćepanović, S., & Quercia, D. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3757482 · Cited by 8 · see citing works
AINeedsPlanner: A Workbook to Support Effective Collaboration Between AI Experts and Clients
Kim, D. H., Shin, H., Yadgarova, S., Son, J., Subramonyam, H., & Kim, J. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661577 · Cited by 7 · see citing works
Wizard of Errors: Introducing and Evaluating Machine Learning Errors in Wizard of Oz Studies
Jansen, A., & Colombo, S. (2022) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3491101.3519684 · Cited by 6 · see citing works
Heo, J., & Lee, U. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3610189 · Cited by 5 · see citing works
Kontogiorgos, D. (2022) · doi:10.31237/osf.io/fpts4 · Cited by 4 · see citing works
Create Effective and Responsible AI User Experiences with The Human-AI Experience (HAX) Toolkit
Vorvoreanu, M. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3574191 · Cited by 3 · see citing works
Krysztoforska, M., Docherty, N., & Biega, A. J. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580717 · Cited by 2 · see citing works
Inoue, S., Nguyen, M.-T., Mizokuchi, H., Nguyen, T.-A., Nguyen, H.-H., & Le, D. (2023) · Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2023.emnlp-industry.49 · Cited by 2 · see citing works
Digitalizing Grief: A Filipino Catholic Perspective
Ocampo, L.-M. A. R. (2025) · Pastoral Psychology · doi:10.1007/s11089-025-01236-0 · Cited by 2 · see citing works
Enhancing intention understanding in voice assistants for older adults using large language models
Fu, J., Kang, C., Qian, Z., & Zhou, Y. (2026) · Behaviour and Information Technology · doi:10.1080/0144929x.2026.2615038 · Cited by 1 · see citing works
Lindvall, M. (2021) · Linköping studies in science and technology. Dissertations · doi:10.3384/diss.diva-176117 · Cited by 1 · see citing works
Saari, L. M., Räikkönen, M., Hytönen, E., & Valkokari, K. (2023) · Systems Engineering · doi:10.1002/sys.21713 · Cited by 1 · see citing works
Fahle, S., Kuhlenkötter, B., Bülow, F., Thewes, F., Prinz, C., & Meske, C. (2022) · SSRN Electronic Journal · doi:10.2139/ssrn.4072322 · Cited by 1 · see citing works
Conversational Assistants and their Applications in Health and Nephrology
Furtado, E. S., Oliveira, F., & Pinheiro, V. C. M. (2022) · Springer eBooks · doi:10.1007/978-3-031-11570-7_18
Konversationelle Assistenten und ihre Anwendungen in Gesundheit und Nephrologie
Furtado, E. S., Oliveira, F., & Pinheiro, V. C. M. (2024) · Springer eBooks · doi:10.1007/978-3-031-65236-3_18
Desirability vs. feasibility: a research through design inquiry of explainable AI
Corti, L., & Yang, J. (2025) · Elsevier eBooks · doi:10.1016/b978-0-44-340553-2.00010-1
ChatGPT vs Gemini: Which Digs Deeper into Arabic Semantics?
Zaimah, N. R., Rizqiyah, C., Hadi, S., Muthiah, R., & Putri, W. N. (2026) · Mantiqu Tayr Journal of Arabic Language · doi:10.25217/mantiqutayr.v6i1.7146 · Cited by 1 · see citing works
Edén, A. S., Faraon, M., & Rönkkö, K. (2026) · International Conference on Tangible, Embedded, and Embodied Interaction (TEI) · doi:10.1145/3731459.3773314
Reinmund, T., Kunze, L., & Jirotka, M. D. (2026) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3772318.3790547
- Authors
- Hong, M. K., Fourney, A., DeBellis, D., & Amershi, S.
- Affiliations
- University of Washington (US)
- Microsoft (United States) (US)
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-11
- OpenAlex
- W3161120666
- Cited by
- 54
- References
- 30
- Citations by year
- 2026: 4 · 2025: 7 · 2024: 12 · 2023: 18 · 2022: 12 · 2021: 1
- FWCI
- 8.03
- Type
- conference paper
- Language
- en
- Open access
- closed
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Keywords
- interactive design tool, boundary objects, user experience design, interdisciplinary collaboration, product teams
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
