Re-examining Whether, Why, and How Human-AI Interaction Is Uniquely Difficult to Design
Open accessThe DOI matches its Crossref record; 50 of the 52 references it cites were found in a registry or an index.
Abstract
Artificial Intelligence (AI) plays an increasingly important role in improving HCI and user experience. Yet many challenges persist in designing and innovating valuable human-AI interactions. For example, AI systems can make unpredictable errors, and these errors damage UX and even lead to undesired societal impact. However, HCI routinely grapples with complex technologies and mitigates their unintended consequences. What makes AI different? What makes human-AI interaction appear particularly difficult to design? This paper investigates these questions. We synthesize prior research, our own design and research experience, and our observations when teaching human-AI interaction. We identify two sources of AI's distinctive design challenges: 1) uncertainty surrounding AI's capabilities, 2) AI's output complexity, spanning from simple to adaptive complex. We identify four levels of AI systems. On each level, designers encounter a different subset of the design challenges. We demonstrate how these findings reveal new insights for designers, researchers, and design tool makers in productively addressing the challenges of human-AI interaction going forward.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Harnessing LMs' generative capabilities to power interaction designs begins with a holistic understanding of these capabilities [5,68]; this includes understanding what LMs can and cannot do under diverse interaction contexts.”
From CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities (2022) · cites this paper as [68]
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Due to the unique characteristics of AI systems (e.g., unpredictability and inconsistency), they are restricted to adapting and applying traditional user experience design principles to AI-powered systems [29].”
From Design implications of generative AI systems for visual storytelling for young learners (2023) · cites this paper as [29]
Experimental — accuracy not yet measuredIn the citing paper
“These challenges include AI's ability to understand complex language, grasp contextual nuances, and provide transparent explanations for its decisions [109], [110].”
From Ethical Framework for Harnessing the Power of AI in Healthcare and Beyond (2024) · cites this paper as [110]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
(2011) · Cited by 3,330
Kaplan, A., & Haenlein, M. (2019) · Business Horizons · doi:10.1016/j.bushor.2018.08.004 · Cited by 3,140 · cited 1×
“The definitions of AI generally fall into two camps [27, 32, 50].”
BackgroundModel Cards for Model Reporting
Mitchell, M., Wu, S., Zaldivar, A., Barnes, P., Vasserman, L., Hutchinson, B., Spitzer, E., Raji, I. D., & Gebru, T. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287596 · Cited by 2,681 · cited 1×
“Prior research has been addressing these challenges by promoting interaction design guidelines [4, 35].”
MethodGuidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439 · cited 4×
“Prior research has been addressing these challenges by promoting interaction design guidelines [4, 35].”
BackgroundMethodInfluentialResearch through design as a method for interaction design research in HCI
Zimmerman, J., Forlizzi, J., & Evenson, S. (2007) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240704 · Cited by 2,304
Self-citationGodel, Escher, Bach: An Eternal Golden Braid.
Lieberson, J., & Hofstadter, D. R. (1980) · The Journal of Philosophy · doi:10.2307/2025598 · Cited by 1,914
Gaver, W. W. (1991) · Proceedings of the SIGCHI conference on Human factors in computing systems Reaching through technology - CHI '91 · doi:10.1145/108844.108856 · Cited by 1,536 · cited 1×
“Designers need to understand the capabilities and limitations of a technology in order to know the possibilities it offers for design [17].”
BackgroundWizard of Oz Studies in HRI: A Systematic Review and New Reporting Guidelines
Riek, L. (2012) · Journal of Human-Robot Interaction · doi:10.5898/jhri.1.1.riek · Cited by 779
Implicit human computer interaction through context
Schmidt, A. (2000) · Personal Technologies · doi:10.1007/bf01324126 · Cited by 637 · cited 1×
“Research has chosen a number of different frames for investigating these challenges including human-AI interaction . design, AI/machine learning as a design material, the design of intelligent systems, designing for/with data [6, 14, 37], and many more [33, 42, 43].”
BackgroundWhy and why not explanations improve the intelligibility of context-aware intelligent systems
Lim, B. Y., Dey, A. K., & Avrahami, D. (2009) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1518701.1519023 · Cited by 552 · cited 1×
“design, AI/machine learning as a design material, the design of intelligent systems, designing for/with data [6, 14, 37], and many more [33, 42, 43].”
BackgroundUX Design Innovation: Challenges for Working with Machine Learning as a Design Material
Dove, G., Halskov, K., Forlizzi, J., & Zimmerman, J. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025739 · Cited by 391 · cited 10×
“This synthesis identified Figure 1: Mapping the human-AI interaction design challenges in the literature [58, 13, 26, 53] onto a user-centered design process (Double Diamond [10]) two sources of AI’s design complexities, and a framework that unravels their effects on design processes.”
BackgroundMethodInfluentialSelf-citationA Collection of Definitions of Intelligence
Legg, S., & Hütter, M. (2007) · arXiv (Cornell University) · doi:10.48550/arxiv.0706.3639 · Cited by 382
Yang, Q., Steinfeld, A., & Zimmerman, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300468 · Cited by 247 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationProceedings of the 2019 on Designing Interactive Systems Conference
(2019) · doi:10.1145/3322276 · Cited by 232
Suede: a Wizard of Oz prototyping tool for speech user interfaces
Klemmer, S. R., Sinha, A. K., Chen, J., Landay, J. A., Aboobaker, N., & Wang, A. (2000) · Proceedings of the 13th annual ACM symposium on User interface software and technology - UIST '00 · doi:10.1145/354401.354406 · Cited by 230 · cited 1×
“One approach is to create Wizard of Oz systems or rule-based simulators as an early-stage interactive AI prototype (e.g. as in [11, 30, 40, 44]).”
MethodInvestigating How Experienced UX Designers Effectively Work with Machine Learning
Yang, Q., Scuito, A., Zimmerman, J., Forlizzi, J., & Steinfeld, A. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196730 · Cited by 226 · cited 2×
“The second project focused on experienced UX practitioners [54].”
BackgroundSelf-citationDesign as bricolage: anthropology meets design thinking
Louridas, P. (1999) · Design Studies · doi:10.1016/s0142-694x(98)00044-1 · Cited by 199 · cited 1×
“Engineers create new technological capabilities; designers create new, valuable products and services with existing technological capabilities and limitations [34].”
BackgroundTalking to Bots: Symbiotic Agency and the Case of Tay
Neff, G. (2016) · International journal of communication · Cited by 196
Using Artificial Intelligence to Augment Human Intelligence
Carter, S., & Nielsen, M. A. (2017) · Distill · doi:10.23915/distill.00009 · Cited by 167
Intelligence on tap: artificial intelligence as a new design material
Holmquist, L. E. (2017) · interactions · doi:10.1145/3085571 · Cited by 159
Grounding Interactive Machine Learning Tool Design in How Non-Experts Actually Build Models
Yang, Q., Suh, J., Chen, N.-C., & Ramos, G. (2018) · Proceedings of the 2018 Designing Interactive Systems Conference · doi:10.1145/3196709.3196729 · Cited by 156 · cited 2×
“The first project focused on novice AI product designers [56].”
BackgroundSelf-citationHuman-Centred Machine Learning
Gillies, M. F. P., Fiebrink, R. A., Tanaka, A., Garcia, J., Bevilacqua, F. P., Héloir, A., Nunnari, F., Mackay, W. E., Amershi, S., Lee, B., et al. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2856492 · Cited by 152 · cited 2×
“Interestingly, the research shows something else, that HCI designers largely struggle to envision and prototype AI systems [2, 3, 13, 18, 22, 26, 49].”
BackgroundField trial of Tiramisu: crowd-sourcing bus arrival times to spur co-design
Zimmerman, J., Tomasic, A., Garrod, C., Yoo, D., Hiruncharoenvate, C., Aziz, R., Thiruvengadam, N. R., Huang, Y., & Steinfeld, A. (2011) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979187 · Cited by 150 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationData, Design and Civics: An Exploratory Study of Civic Tech
Boehner, K., & DiSalvo, C. F. (2016) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858326 · Cited by 138 · cited 2×
“This body of work encourages designers to investigate the lived-life of data and discover AI design opportunities [6, 7, 14].”
BackgroundFeinberg, M. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025837 · Cited by 115 · cited 2×
“This body of work encourages designers to investigate the lived-life of data and discover AI design opportunities [6, 7, 14].”
BackgroundCalendar.help: Designing a Workflow-Based Scheduling Agent with Humans in the Loop
Cranshaw, J., Elwany, E., Newman, T., Kocielnik, R., Yu, B., Soni, S., Teevan, J., & Monroy-Hernández, A. (2017) · Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025780 · Cited by 106
Yang, Q., Zimmerman, J., Steinfeld, A., Carey, L., & Antaki, J. F. (2016) · Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems · doi:10.1145/2858036.2858373 · Cited by 105 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationConnected Baby Bottle: A Design Case Study Towards a Framework for Data-Enabled Design
Bogers, S., Frens, J. W., van Kollenburg, J., Deckers, E., & Hummels, C. (2016) · Designing Interactive Systems Conference · doi:10.1145/2901790.2901855 · Cited by 100 · cited 2×
“This body of work encourages designers to investigate the lived-life of data and discover AI design opportunities [6, 7, 14].”
BackgroundMethodSketching NLP: A Case Study of Exploring the Right Things To Design with Language Intelligence
Yang, Q., Cranshaw, J., Amershi, S., Iqbal, S. T., & Teevan, J. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300415 · Cited by 90 · cited 4×
“However, this approach fails to address any of the UX issues that will come from AI inference errors because there is no way to simulate these errors [52].”
BackgroundInfluentialSelf-citationMoving beyond user-centered design
Forlizzi, J. (2018) · interactions · doi:10.1145/3239558 · Cited by 80 · cited 1×
“…researchers have proposed that AI may require design processes less focused on one group of users, and instead on many user groups and stakeholders [15]; processes focused less on fast, iterative prototyping, and instead on existing datasets and functioning AI systems [51]; or processes focused…”
BackgroundYang, Q., Steinfeld, A. M., & Zimmerman, J. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1904.09612 · Cited by 80 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationPrototyping Social Interaction
Kurvinen, E., Koskinen, I., & Battarbee, K. (2008) · Design Issues · doi:10.1162/desi.2008.24.3.46 · Cited by 73 · cited 1×
“For example, CSCW research has investigated how to prototype workplace knowledge sharing systems whose af-fordance co-evolves with its users’ behaviors, the interactions among its users, and the organizational contexts at large [31].”
BackgroundOn the Design of OLO Radio: Investigating Metadata as a Design Material
Odom, W., & Duel, T. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3173678 · Cited by 71 · cited 2×
“For example, [37] investigated users’ music app metadata as a material for designing contemplative music experience; [24] explored the design opportunities around intimate somatic data.”
BackgroundDesigning with Data: Improving the User Experience with A/B Testing
King, R., Churchill, E. F., & Tan, C. (2017) · Cited by 70
Calendar.Help: Designing a Workflow-Based Scheduling Agent with Humans in the LoopPreprint
Cranshaw, J., Kocielnik, R., Yu, B., & Teevan, J. B. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2940295 · Cited by 65
Planning Adaptive Mobile Experiences When Wireframing
Yang, Q., Zimmerman, J., Steinfeld, A., & Tomasic, A. (2016) · Proceedings of the 2016 ACM Conference on Designing Interactive Systems · doi:10.1145/2901790.2901858 · Cited by 55 · cited 5×
“This synthesis identified Figure 1: Mapping the human-AI interaction design challenges in the literature [58, 13, 26, 53] onto a user-centered design process (Double Diamond [10]) two sources of AI’s design complexities, and a framework that unravels their effects on design processes.”
BackgroundMethodInfluentialSelf-citationRADAR: A Personal Assistant that Learns to Reduce Email Overload
Freed, M. C., Carbonell, J. G., Gordon, G. J., Hayes, J., Myers, B. A., Siewiorek, D. P., Smith, S. F., Steinfeld, A. M., & Tomasic, A. (2018) · doi:10.1184/r1/6625490 · Cited by 53
Self-citationPrototyping ways of prototyping AI
van Allen, P. (2018) · interactions · doi:10.1145/3274566 · Cited by 53 · cited 5×
“To make prototyping human-AI interaction easier, researchers have created simple-rule-based simulators [49, 7]) as AI prototyping tools.”
BackgroundMethodInfluentialHelms, K. (2019) · Designing Interactive Systems Conference · doi:10.1145/3322276.3322290 · Cited by 50 · cited 1×
“For example, [37] investigated users’ music app metadata as a material for designing contemplative music experience; [24] explored the design opportunities around intimate somatic data.”
BackgroundLearning patterns of pick-ups and drop-offs to support busy family coordination
Davidoff, S., Ziebart, B. D., Zimmerman, J., & Dey, A. K. (2011) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1978942.1979119 · Cited by 44 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationKayacik, C., Chen, S., Noerly, S., Holbrook, J., Roberts, A., & Douglas Eck, C. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3299059 · Cited by 38 · cited 1×
“Some designers also found it challenging to effectively collaborate with AI engineers, because they lacked a shared workflow, boundary objects, or a common language for scaffolding the collaboration [19, 28, 52].”
BackgroundDesign through matchmaking: technology in search of users
Bly, S., & Churchill, E. F. (1999) · Interactions · doi:10.1145/296165.296174 · Cited by 34 · cited 2×
“Even when Figure 2: Mapping UX design challenges of AI in prior research on a technology-driven design innovation process [41, 5] designers understand how AI works, they often found it difficult to ideate many possible new interactions and novel experiences with much fluidity [13, 58].”
BackgroundExamining Difficulties Software Developers Encounter in the Adoption of Statistical Machine Learning
Patel, K., Fogarty, J. A., Landay, J. A., & Harrison, B. L. (2008) · Cited by 32
Artificial Intelligence and Life
Skyttner, L. G. (1996) · doi:10.1007/978-1-349-13532-5_6 · Cited by 31
When User Experience Designers Partner with Data Scientists.
Girardin, F., & Lathia, N. (2017) · National Conference on Artificial Intelligence · Cited by 29
Vio: a mixed-initiative approach to learning and automating procedural update tasks
Zimmerman, J., Tomasic, A., Simmons, I., Hargraves, I. G., Mohnkern, K., Cornwell, J. B., & McGuire, R. M. (2007) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/1240624.1240843 · Cited by 25 · cited 1×
“First, we draw on our many years of experience in designing a wide range of AI systems, from simple adaptive user interfaces [58, 12], to large-scale crowd-sourced transportation systems [59]; from clinical decision supports [57, 55] to natural language productivity tools [52, 60, 16].”
BackgroundSelf-citationDesigning natural speech interactions for the living room
Stifelman, L., Elman, A., & Sullivan, A. (2013) · CHI '13 Extended Abstracts on Human Factors in Computing Systems · doi:10.1145/2468356.2468574 · Cited by 13 · cited 1×
“One approach is to create Wizard of Oz systems or rule-based simulators as an early-stage interactive AI prototype (e.g. as in [11, 30, 40, 44]).”
MethodEmbedding User Understanding in the Corporate Culture
Stone, M. V., Bentley, F. R., White, B., & Shebanek, M. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2851592 · Cited by 12 · cited 1×
“Its breakdowns can be especially harmful for under-served user populations, including people with disabilities [45].”
BackgroundBuilding Adaptive Accessible Context-Aware for User Interface Tailored to Disable Users
Zouhaier, L., Hlaoui, Y. B., & Ayed, L. J. B. (2013) · IEEE Annual Computer Software and Applications Conference Workshops (COMPSACW) · doi:10.1109/compsacw.2013.39 · Cited by 11
Rapid prototyping IoT solutions based on Machine Learning
Rizzo, A., Montefoschi, F., Caporali, M., Gisondi, A., Burresi, G., & Giorgi, R. (2017) · European Conference on Cognitive Ergonomics · doi:10.1145/3121283.3121291 · Cited by 9 · cited 3×
“Nor can they easily prototype the types of inference errors a not-yet developed AI system might make [29, 42, 49].”
Background
Research works printed but not linked: 9
- 2017. Designing the User Experience of Machine Learning Systems Papers from the 2017 AAAI Spring Symposium Technical Report SS-17-04 Palo Alto California USA March 27--29 2017. AAAI. https://mikek-parc.github.io/AAAI-UX-ML/
- 2018. The Design of the User Experience for Artificial Intelligence (The UX of AI), Papers from the 2018 AAAI Spring Symposium, Palo Alto, California, USA, March 26--28, 2018. AAAI. https://mikek-parc.github.io/AAAI-UX-AI/
- Amber Cartwright. 2016. Invisible Design: Co-Designing with Machines. http://airbnb.design/invisible-design/
- Design Council. 2005. The 'double diamond' design process model. Design Council (2005).
- Patrick Hebron. 2016. Machine learning for designers. O'Reilly Media.
- 10.1145/354401.354406 · DOI · Paper page
- Kayur Dushyant Patel. 2012. Lowering the Barrier to Applying Machine Learning. Ph.D. Dissertation. University of Washington.
- Qian Yang, Nikola Banovic, and John Zimmerman. 2018. Mapping Machine Learning Advances from HCI Research to Reveal Starting Places for Design Research. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems - CHI '18 (CHI '18). ACM.
- 10.1145/3196709.3196730 · DOI · Paper page
Cited by
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J. C., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · arXiv · doi:10.48550/arxiv.2304.03442 · Cited by 2,173
Generative Agents: Interactive Simulacra of Human Behavior
Park, J. S., O'Brien, J., Cai, C. J., Morris, M. R., Liang, P., & Bernstein, M. S. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606763 · Cited by 2,187
“However, in other application domains, it will be important to follow best practices in human-AI design [5, 107] to understand errors and how they might percolate into the user experience.”
BackgroundDesign Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2021) · arXiv · doi:10.48550/arxiv.2109.06977 · Cited by 635
Design Guidelines for Prompt Engineering Text-to-Image Generative Models
Liu, V., & Chilton, L. B. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501825 · Cited by 639
Six Human-Centered Artificial Intelligence Grand Challenges
Ozmen Garibay, O., Winslow, B., Andolina, S., Antona, M., Bodenschatz, A., Coursaris, C., Falco, G., Fiore, S. M., Garibay, I., Grieman, K., et al. (2023) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2022.2153320 · Cited by 501
“Recent research already reported that many challenges have been encountered in the process of developing HCAI systems (Yang et al., 2020).”
Metadata
- Authors
- Yang, Q., Steinfeld, A., Rosé, C., & Zimmerman, J.
- Affiliations
- Carnegie Mellon University (US)
- Venue
- Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2020-04-21
- Pages
- 1-13
- OpenAlex
- W3032875465
- Semantic Scholar
- 529025645c70a935221bd434484faee695ad0f25
- Cited by
- 622
- References
- 52
- Citations by year
- 2026: 120 · 2025: 134 · 2024: 115 · 2023: 108 · 2022: 79 · 2021: 56 · 2020: 9
- FWCI
- 46.02
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- Funders
- The National Institute on Disability Independent Living and Rehabilitation Research (NIDILRR) (90REGE0007)
- Topics
- Physical Sciences › Computer Science › Human-Computer Interaction › Innovative Human-Technology Interaction
- Physical Sciences › Computer Science › Human-Computer Interaction › Usability and User Interface Design
- Social Sciences › Psychology › Social Psychology › Social Robot Interaction and HRI
- Keywords
- human-AI interaction, AI system design, design challenges, unintended consequences of AI, user experience, human-computer interaction design
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.