CoChat
Cited by 162

The DOI matches its Crossref record, with a small difference in title or year; 54 of the 54 references it cites were found in a registry or an index.

Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs

Subramonyam, Hari, Pea, Roy, Pondoc, Christopher, Agrawala, Maneesh, Seifert, Colleen · 2024 · Proceedings of the CHI Conference on Human Factors in Computing Systems

Abstract

Large language models (LLMs) exhibit dynamic capabilities and appear to comprehend complex and ambiguous natural language prompts. However, calibrating LLM interactions is challenging for interface designers and end-users alike. A central issue is our limited grasp of how human cognitive processes begin with a goal and form intentions for executing actions, a blindspot even in established interaction models such as Norman’s gulfs of execution and evaluation. To address this gap, we theorize how end-users ‘envision’ translating their goals into clear intentions and craft prompts to obtain the desired LLM response. We define a process of Envisioning by highlighting three misalignments on not knowing: (1) what the task should be, (2) how to instruct the LLM to do the task, and (3) what to expect for the LLM’s output in meeting the goal. Finally, we make recommendations to narrow the gulf of envisioning in human-LLM interactions.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3613904.3642754

Cites

54 of 54 references verified0 citation errors · 0 not checkable

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Show all 54 references

Research works printed but not linked: 141

  1. 10.1007/s00778-019-00567-8 · DOI · Paper page
  2. 10.1145/882262.882352 · DOI · Paper page
  3. Tyler Angert, Miroslav Ivan Suzara, Jenny Han, Christopher Lawrence Pondoc, and Hariharan Subramonyam. 2023. Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts. arXiv preprint arXiv:2308.03921 (2023). · arXiv
  4. Franscesca Bacci, Federico Maria Cau, and Lucio Davide Spano. 2020. Inspecting data using natural language queries. In Computational Science and Its Applications–ICCSA 2020: 20th International Conference, Cagliari, Italy, July 1–4, 2020, Proceedings, Part VI 20. Springer, 771–782.
  5. 10.2307/3192667 · DOI
  6. Yavar Bathaee. 2017. The artificial intelligence black box and the failure of intent and causation. Harv. JL & Tech. 31 (2017), 889.
  7. 10.1016/S0020-7373(84)80017-6 · DOI
  8. Olav W Bertelsen and Susanne Bødker. 2003. Activity theory. HCI models, theories, and frameworks: Toward a multidisciplinary science (2003), 291–324.
  9. Bernd Bohnet Vinh Tran Pat Verga Roee Aharoni Daniel Andor Livio Baldini Soares Massimiliano Ciaramita Jacob Eisenstein Kuzman Ganchev Jonathan Herzig Kai Hui Tom Kwiatkowski Ji Ma Jianmo Ni Tal Schuster Lierni Sestorain Saralegui William Weston Cohen Michael Collins Dipanjan Das Don Metzler Slav Petrov and Kellie Webster. 2022. Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models. https://arxiv.org/abs/2212.08037 · arXiv
  10. Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Advances in neural information processing systems 29 (2016).
  11. 10.1007/978-3-319-60642-2_2 · DOI
  12. 10.1016/j.pragma.2009.12.012 · DOI · Paper page
  13. Ann L Brown. 2017. Metacognitive development and reading. In Theoretical issues in reading comprehension. Routledge, 453–482.
  14. 10.1126/science.aal4230 · DOI · Paper page
  15. John M Carroll and Judith Reitman Olson. 1988. Mental models in human-computer interaction. Handbook of human-computer interaction (1988), 45–65.
  16. Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, and Dylan Hadfield-Menell. 2023. Explore, Establish, Exploit: Red Teaming Language Models from Scratch. arXiv preprint arXiv:2306.09442 (2023). · arXiv
  17. Jiuhai Chen Lichang Chen Heng Huang and Tianyi Zhou. 2023. When do you need Chain-of-Thought Prompting for ChatGPT?arxiv:2304.03262 [cs.AI] · arXiv
  18. Xiang’Anthony’ Chen, Jeff Burke, Ruofei Du, Matthew K Hong, Jennifer Jacobs, Philippe Laban, Dingzeyu Li, Nanyun Peng, Karl DD Willis, Chien-Sheng Wu, 2023. Next Steps for Human-Centered Generative AI: A Technical Perspective. arXiv preprint arXiv:2306.15774 (2023). · arXiv
  19. Richard E Clark, David F Feldon, Jeroen JG Van Merrienboer, Kenneth A Yates, and Sean Early. 2008. Cognitive task analysis. In Handbook of research on educational communications and technology. Routledge, 577–593.
  20. Nigel Cross. 2001. Design cognition: Results from protocol and other empirical studies of design activity. Design knowing and learning: Cognition in design education (2001), 79–103.
  21. Mihaly Csikszentmihalyi and Jacob W Getzels. 1971. Discovery-oriented behavior and the originality of creative products: A study with artists.Journal of personality and social psychology 19, 1 (1971), 47.
  22. Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. 2023. Drive as you speak: Enabling human-like interaction with large language models in autonomous vehicles. arXiv preprint arXiv:2309.10228 (2023). · arXiv
  23. Victor Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. arXiv preprint arXiv:2303.02927 (2023). · arXiv
  24. Danica Dillion, Niket Tandon, Yuling Gu, and Kurt Gray. 2023. Can AI language models replace human participants?Trends in Cognitive Sciences (2023).
  25. 10.1016/j.pragma.2022.03.001 · DOI
  26. Kees Dorst. 2011. The core of ‘design thinking’and its application. Design studies 32, 6 (2011), 521–532.
  27. Kees Dorst and Nigel Cross. 2001. Creativity in the design process: co-evolution of problem–solution. Design studies 22, 5 (2001), 425–437.
  28. Karl Duncker. 1945. On problem-solving.(Psychological Monographs, No. 270.). (1945).
  29. 10.1111/1540_6245.jaac21.3.0283 · doi:10.1111/1540_6245.jaac21.3.0283
  30. 10.1109/MCG.2015.91 · DOI
  31. Umer Farooq and Jonathan Grudin. 2016. Human-computer integration. interactions 23, 6 (2016), 26–32.
  32. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models. arXiv preprint arXiv:2305.08283 (2023). · arXiv
  33. 10.1145/3544549.3585737 · DOI
  34. Asbjørn Følstad and Petter Bae Brandtzæg. 2017. Chatbots and the new world of HCI. interactions 24, 4 (2017), 38–42.
  35. Chris Frith and Uta Frith. 2005. Theory of mind. Current biology 15, 17 (2005), R644–R645.
  36. Yao Fu Hao Peng Ashish Sabharwal Peter Clark and Tushar Khot. 2023. Complexity-Based Prompting for Multi-Step Reasoning. arxiv:2210.00720 [cs.CL] · arXiv
  37. 10.1145/2807442.2807478 · DOI · Paper page
  38. James J Gibson. 1977. The theory of affordances. Hilldale, USA 1, 2 (1977), 67–82.
  39. Peter M Gollwitzer and Gabriele Oettingen. 2020. Implementation intentions. In Encyclopedia of behavioral medicine. Springer, 1159–1164.
  40. Charles Goodwin and John Heritage. 1990. Conversation analysis. Annual review of anthropology 19, 1 (1990), 283–307.
  41. Herbert P Grice. 1975. Logic and conversation. In Speech acts. Brill, 41–58.
  42. Joy Paul Guilford. 1956. The structure of intellect.Psychological bulletin 53, 4 (1956), 267.
  43. Andrew B Hargadon and Beth A Bechky. 2006. When collections of creatives become creative collectives: A field study of problem solving at work. Organization science 17, 4 (2006), 484–500.
  44. John R Hayes. 2013. A new framework for understanding cognition and affect in writing. In The science of writing. Routledge, 1–27.
  45. 10.1177/1541931218621047 · DOI
  46. Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021. Surface form competition: Why the highest probability answer isn’t always right. arXiv preprint arXiv:2104.08315 (2021). · arXiv
  47. 10.1109/TVCG.2017.2744684 · DOI
  48. 10.1145/3025453.3025765 · DOI · Paper page
  49. Thomas J Howard, Stephen J Culley, and Elies Dekoninck. 2008. Describing the creative design process by the integration of engineering design and cognitive psychology literature. Design studies 29, 2 (2008), 160–180.
  50. Edwin Hutchins. 1987. Metaphors for interface design. Institute for Cognitive Science, University of California, San Diego.
  51. Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.
  52. Michael Jackson. 1995. Software Requirements & Specifications: a lexicon of practice, principles and prejudices. ACM Press/Addison-Wesley Publishing Co.
  53. David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11.
  54. 10.1145/3571730 · DOI · Paper page
  55. Peiling Jiang, Jude Rayan, Steven P Dow, and Haijun Xia. 2023. Graphologue: Exploring Large Language Model Responses with Interactive Diagrams. arXiv preprint arXiv:2305.11473 (2023). · arXiv
  56. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and Applications of Large Language Models. arXiv preprint arXiv:2307.10169 (2023). · arXiv
  57. 10.1145/3170427.3188445 · DOI
  58. Yoonsu Kim, Jueon Lee, Seoyoung Kim, Jaehyuk Park, and Juho Kim. 2023. Understanding Users’ Dissatisfaction with ChatGPT Responses: Types, Resolving Tactics, and the Effect of Knowledge Level. arXiv preprint arXiv:2311.07434 (2023). · arXiv
  59. 10.1109/VLHCC.2004.47 · DOI · Paper page
  60. Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, 2023. ChatGPT: Jack of all trades, master of none. Information Fusion (2023), 101861.
  61. Shunsuke Koga. 2023. Exploring the Pitfalls of Large Language Models: Inconsistency and Inaccuracy in Answering Pathology Board Examination-Style Questions. medRxiv (2023), 2023–08.
  62. 10.1017/jme.2022.13 · DOI
  63. 10.1145/3526113.3545693 · DOI
  64. 10.1037/0278-7393.18.2.391 · DOI
  65. Brenden M. Lake Tal Linzen and Marco Baroni. 2019. Human few-shot learning of compositional instructions. arxiv:1901.04587 [cs.CL] · arXiv
  66. 10.1145/2470654.2481301 · DOI
  67. Karl Spencer Lashley 1951. The problem of serial order in behavior. Vol. 21. Bobbs-Merrill Oxford.
  68. Jiachang Liu Dinghan Shen Yizhe Zhang Bill Dolan Lawrence Carin and Weizhu Chen. 2021. What Makes Good In-Context Examples for GPT-3?arxiv:2101.06804 [cs.CL] · arXiv
  69. Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi and Graham Neubig. 2021. Pre-train Prompt and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. arxiv:2107.13586 [cs.CL] · arXiv
  70. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860 (2023). · arXiv
  71. 10.1016/j.cognition.2004.12.009 · DOI · Paper page
  72. Joseph B Lyons, Sean Mahoney, Kevin T Wynne, and Mark A Roebke. 2018. Viewing machines as teammates: A qualitative study. In 2018 AAAI Spring Symposium Series.
  73. Norman RF Maier. 1931. Reasoning in humans. II. The solution of a problem and its appearance in consciousness.Journal of comparative Psychology 12, 2 (1931), 181.
  74. 10.1007/978-3-030-22868-2_90 · DOI
  75. 10.1145/356835.356841 · DOI · Paper page
  76. 10.18653/v1 · doi:10.18653/v1
  77. George A Miller. 1956. The magical number seven, plus or minus two: Some limits on our capacity for processing information.Psychological review 63, 2 (1956), 81.
  78. George A Miller, Galanter Eugene, and Karl H Pribram. 2017. Plans and the Structure of Behaviour. In Systems Research for Behavioral Science. Routledge, 369–382.
  79. 10.1145/3387166 · DOI · Paper page
  80. 10.1016/S0020-7373(87)80020-2 · DOI
  81. Robert Munro, Steven Bethard, Victor Kuperman, Vicky Tzuyin Lai, Robin Melnick, Christopher Potts, Tyler Schnoebelen, and Harry Tily. 2010. Crowdsourcing and language studies: the new generation of linguistic data. In Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk. Association for Computational Linguistics, Los Angeles, 122–130. https://aclanthology.org/W10-0719
  82. Dennis C Neale and John M Carroll. 1997. The role of metaphors in user interface design. In Handbook of human-computer interaction. Elsevier, 441–462.
  83. 10.1126/science.134.3495.2011 · DOI · Paper page
  84. Allen Newell and Herbert A Simon. 2007. Computer science as empirical inquiry: Symbols and search. In ACM Turing award lectures. 1975.
  85. Allen Newell, Herbert Alexander Simon, 1972. Human problem solving. Vol. 104. Prentice-hall Englewood Cliffs, NJ.
  86. 10.1201/b15703 · DOI · Paper page
  87. Donald A Norman. 2014. Some observations on mental models. In Mental models. Psychology Press, 15–22.
  88. David Oniani, Jordan Hilsman, Yifan Peng, Ronald K Poropatich, COL Pamplin, LTC Legault, Yanshan Wang, 2023. From Military to Healthcare: Adopting and Expanding Ethical Principles for Generative Artificial Intelligence. arXiv preprint arXiv:2308.02448 (2023). · arXiv
  89. 10.1145/3491102.3517739 · DOI
  90. 10.1145/3318464.3383128 · DOI
  91. 10.1006/cogp.1997.0655 · DOI
  92. Roy D Pea. 1982. What is planning development the development of?New Directions for Child and Adolescent Development 1982, 18 (1982), 5–27.
  93. Ben Prystawski Paul Thibodeau Christopher Potts and Noah D. Goodman. 2023. Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models. arxiv:2209.08141 [cs.CL] · arXiv
  94. A Terry Purcell and John S Gero. 1996. Design and other types of fixation. Design studies 17, 4 (1996), 363–383.
  95. Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2023. Measuring attribution in natural language generation models. Computational Linguistics (2023), 1–66.
  96. Byron Reeves and Clifford Nass. 1996. The media equation: How people treat computers, television, and new media like real people. Cambridge, UK 10, 10 (1996).
  97. 10.1504/JDR.2004.009842 · DOI
  98. Arnold Sameroff. 2009. The transactional model.American Psychological Association.
  99. Roger C Schank and Robert P Abelson. 2013. Scripts, plans, goals, and understanding: An inquiry into human knowledge structures. Psychology Press.
  100. D Schon. 1983. Becoming a reflective practitioner. How professionals think in action. London: Temple Smith (1983).
  101. Gregory Schraw and Rayne Sperling Dennison. 1994. Assessing metacognitive awareness. Contemporary educational psychology 19, 4 (1994), 460–475.
  102. John R Searle, Ferenc Kiefer, Manfred Bierwisch, 1980. Speech act theory and pragmatics. Vol. 10. Springer.
  103. Colleen M Seifert David E Meyer Natalie Davidson Andrea L Patalano and Ilan Yaniv. 1994. Demystification of cognitive insight: Opportunistic assimilation and the prepared-mind hypothesis. (1994). · DOI · Paper page
  104. 10.1111/1467-8721.00148 · DOI
  105. 10.1145/2984511.2984588 · DOI · Paper page
  106. 10.1145/3491102.3501972 · DOI
  107. 10.18653/v1 · doi:10.18653/v1
  108. Leixian Shen, Enya Shen, Yuyu Luo, Xiaocong Yang, Xuming Hu, Xiongshuai Zhang, Zhiwei Tai, and Jianmin Wang. 2022. Towards natural language interfaces for data visualization: A survey. IEEE transactions on visualization and computer graphics (2022).
  109. 10.18653/v1 · doi:10.18653/v1
  110. 10.1080/01449298208914450 · DOI · Paper page
  111. Herbert A Simon. 1956. Rational choice and the structure of the environment.Psychological review 63, 2 (1956), 129.
  112. Herbert Alexander Simon. 1997. Models of bounded rationality: Empirically grounded economic reason. Vol. 3. MIT press.
  113. 10.1109/TVCG.2020.2978050 · DOI
  114. 10.1109/MCG.2020.2986902 · DOI
  115. 10.1006/imms.1993.1028 · DOI · Paper page
  116. 10.1017/s0140525x00003435 · DOI · Paper page
  117. Matthew Stone. 2005. Communicative intentions and conversational processes in humanhuman and human-computer dialogue. Approaches to studying world-situated language use (2005), 39–70.
  118. 10.1145/3491102.3517537 · DOI
  119. 10.1145/3242587.3242618 · DOI
  120. Hariharan Subramonyam, Colleen Seifert, and Eytan Adar. 2021. Towards a process model for co-creating AI experiences. In Designing Interactive Systems Conference 2021. 1529–1543.
  121. 10.1145/3313831.3376155 · DOI
  122. Alaina N Talboy and Elizabeth Fuller. 2023. Challenging the appearance of machine intelligence: Cognitive bias in LLMs. arXiv preprint arXiv:2304.01358 (2023). · arXiv
  123. Jenifer Tidwell. 2010. Designing interfaces: Patterns for effective interaction design. " O’Reilly Media, Inc.".
  124. Tomer Ullman. 2023. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.08399 (2023). · arXiv
  125. Helena Vasconcelos Gagan Bansal Adam Fourney Q. Vera Liao and Jennifer Wortman Vaughan. 2023. Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions. arxiv:2302.07248 [cs.HC] · arXiv
  126. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arxiv:2203.11171 [cs.CL] · arXiv
  127. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.). Vol. 35. Curran Associates, Inc., 24824–24837. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf
  128. Justin D Weisz, Michael Muller, Jessica He, and Stephanie Houde. 2023. Toward general design principles for generative AI applications. arXiv preprint arXiv:2301.05578 (2023). · arXiv
  129. Christopher D Wickens, Justin G Hollands, Simon Banbury, and Raja Parasuraman. 2015. Engineering psychology and human performance. Psychology Press.
  130. Merlin C Wittrock. 1989. Generative processes of comprehension. Educational psychologist 24, 4 (1989), 345–376.
  131. Larry E Wood. 1997. User interface design: Bridging the gap from user requirements to design. CRC Press.
  132. Austin P Wright, Zijie J Wang, Haekyu Park, Grace Guo, Fabian Sperrle, Mennatallah El-Assady, Alex Endert, Daniel Keim, and Duen Horng Chau. 2020. A comparative analysis of industry human-AI interaction guidelines. arXiv preprint arXiv:2010.11761 (2020). · arXiv
  133. 10.1145/3491102.3517582 · DOI · Paper page
  134. 10.1145/3543873.3587309 · DOI
  135. 10.1145/3196709.3196730 · DOI · Paper page
  136. Shunyu Yao Dian Yu Jeffrey Zhao Izhak Shafran Thomas L. Griffiths Yuan Cao and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. arxiv:2305.10601 [cs.CL] · arXiv
  137. Samuel Yeom, Irene Giacomelli, Matt Fredrikson, and Somesh Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st computer security foundations symposium (CSF). IEEE, 268–282.
  138. Richard M Young. 2014. Surrogates and mappings: Two kinds of conceptual models for interactive devices. In Mental models. Psychology Press, 43–60.
  139. Bowen Yu and Cláudio T Silva. 2019. FlowSense: A natural language interface for visual data exploration within a dataflow system. IEEE transactions on visualization and computer graphics 26, 1 (2019), 1–11.
  140. JD Zamfirescu-Pereira Heather Wei Amy Xiao Kitty Gu Grace Jung Matthew G Lee Bjoern Hartmann and Qian Yang. 2023. Herding AI Cats: Lessons from Designing a Chatbot by Prompting GPT-3. (2023). · DOI
  141. Tony Z. Zhao Eric Wallace Shi Feng Dan Klein and Sameer Singh. 2021. Calibrate Before Use: Improving Few-Shot Performance of Language Models. arxiv:2102.09690 [cs.CL] · arXiv

Cited by

See all 162

Metadata

Authors
Subramonyam, H., Pea, R., Pondoc, C., Agrawala, M., & Seifert, C.
Affiliations
  • Stanford University, United States
  • Graduate School of Education, Stanford University, United States
  • Computer Science, Stanford University, United States
  • Psychology, U Michigan, United States
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-19
Cited by
162
References
54
Funders
NSF (2229873)

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. https://app.cochat.ai/papers/10.1145/3613904.3642754
The paper
Subramonyam, H., Pea, R., Pondoc, C., Agrawala, M., & Seifert, C. (2024). Bridging the Gulf of Envisioning: Cognitive Challenges in Prompt Based Interactions with LLMs. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3642754

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.