The Metacognitive Demands and Opportunities of Generative AI

GenAI systems present usability challenges around prompting, evaluating and relying on outputs, and optimizing workflows.

“Generative AI (GenAI) systems offer unprecedented opportunities for transforming professional and personal work, yet present challenges around prompting, evaluating and relying on outputs, and optimizing workflows.”

• cites [208] 208. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1-21. https://doi.org/10.1145/ 3544548.3581388 sourceNot checked — no full text of the source
• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  The Discussion addresses dogmatism, uncertainty, and information search, not the use of generative AI. Searches for GenAI prompting, output evaluation, and workflow optimization found no passage addressing those challenges in our copy. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [14] 14. Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (April 2023), 78:85-78:111. https://doi.org/10.1145/3586030 sourcePartly supported

  Sections 4.2.3 and 4.2.5 describe difficulties prompting Copilot and validating its suggestions; section 6.2 discusses suggestions that disrupt programmers’ workflow. These findings support the claim for AI-assisted programming, but the paper does not establish them for GenAI systems across professional and personal work generally. The indexed passages provide no page numbers. (LLM, conf=0.86)

  Experimental — accuracy not yet measured
• cites [154] 154. Advait Sarkar. 2023. Exploring Perspectives on the Impact of Artificial In- telligence on the Creativity of Knowledge Work: Beyond Mechanised Pla- giarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK '23). Association for Computing Machinery, New York, NY, USA, 1-17. https: //doi.org/10.1145/3596671.3597650 sourcePartly supported

  The paper identifies difficulty writing effective prompts (Section 7.1, p. 10), evaluating generated code (Section 5, p. 7), and avoiding over-reliance on it (Section 5, p. 7). These findings concern AI-assisted programming; the passages do not establish the same challenges for GenAI systems generally, and the discussion of workflows (Section 7.3, p. 13) describes their use rather than a challenge of optimizing them. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

These same properties pose a challenge for designing GenAI systems to be human-centered.

“However, these same properties also pose a challenge for designing GenAI systems to be human-centered [34].”

• cites [34] 34. Xiang 'Anthony' Chen, Jeff Burke, Ruofei Du, Matthew K. Hong, Jennifer Jacobs, Philippe Laban, Dingzeyu Li, Nanyun Peng, Karl D. D. Willis, Chien-Sheng Wu, and Bolei Zhou. 2023. Next Steps for Human-Centered Generative AI: A Tech- nical Perspective. https://doi.org/10.48550/arXiv.2306.15774 arXiv:2306.15774 [cs]. sourceSupported

User studies reveal a range of usability challenges around prompting, evaluating and relying on outputs, and deciding on an automation strategy.

“User studies reveal a range of usability challenges around prompting [208], evaluating and relying on outputs [156], and deciding on an automation strategy: whether and how to integrate GenAI into workflows [14,154].”

• cites [208] 208. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1-21. https://doi.org/10.1145/ 3544548.3581388 sourceNot checked — no full text of the source
• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  Citation [156] is attached to the claim about evaluating and relying on outputs. The Materials and Methods describe judgments about flickering-dot displays, while the Results and Discussion address confidence and information-seeking decisions. Searches for output evaluation, GenAI, and automation found no passage about usability challenges in evaluating or relying on generated outputs. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [14] 14. Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (April 2023), 78:85-78:111. https://doi.org/10.1145/3586030 sourceSupported
• cites [154] 154. Advait Sarkar. 2023. Exploring Perspectives on the Impact of Artificial In- telligence on the Creativity of Knowledge Work: Beyond Mechanised Pla- giarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK '23). Association for Computing Machinery, New York, NY, USA, 1-17. https: //doi.org/10.1145/3596671.3597650 sourceSupported

The generality of GenAI poses a higher-level metacognitive demand: the challenge of knowing whether and how to incorporate GenAI into workflows.

“Alongside the local interactions with GenAI systems, the generality of GenAI poses another, higherlevel metacognitive demand: the challenge of knowing whether and how to incorporate GenAI into workflows-i.e., one's 'automation strategy' (see also [154]).”

• cites [154] 154. Advait Sarkar. 2023. Exploring Perspectives on the Impact of Artificial In- telligence on the Creativity of Knowledge Work: Beyond Mechanised Pla- giarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK '23). Association for Computing Machinery, New York, NY, USA, 1-17. https: //doi.org/10.1145/3596671.3597650 sourcePartly supported

  Section 7.3 (p. 13) reports that some programmers developed AI-dependent workflows for particular tasks, and Section 8.2 (p. 15) describes the wide range of tasks a prompt can specify. These passages support the idea that programmers make choices about how to use AI assistance, but they do not establish the broader claim that GenAI’s generality creates a higher-level metacognitive demand to decide whether and how to incorporate it into workflows. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Metacognitive abilities can be taught.

“Firstly, given that metacognitive abilities can be taught [45,52,126], we can improve users' metacognition via metacognitive support strategies that can be integrated into GenAI systems.”

• cites [45] 45. Hester de Boer, Anouk S. Donker, Danny D. N. M. Kostons, and Greetje P. C. van der Werf. 2018. Long-term effects of metacognitive strategy instruction on student academic performance: A meta-analysis. Educational Research Review 24 (June 2018), 98-115. https://doi.org/10.1016/j.edurev.2018.03.002 sourceSupported
• cites [52] 52. A. S. Donker, H. de Boer, D. Kostons, C. C. Dignath van Ewijk, and M. P. C. van der Werf. 2014. Effectiveness of learning strategy instruction on academic performance: A meta-analysis. Educational Research Review 11 (Jan. 2014), 1-26. https://doi.org/10.1016/j.edurev.2013.11.002 sourceSupported
• cites [126] 126. Daniel Muijs and Christian Bokhove. 2020. Metacognition and Self-Regulation: Evidence Review. Technical Report. Education Endowment Foundation. https:// eric.ed.gov/?id=ED612286 Publication Title: Education Endowment Foundation ERIC Number: ED612286.Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

Recent HCI work has begun to pursue integrating metacognitive support into GenAI systems, albeit without explicitly grounding it in metacognition.

“Recent HCI work has begun to pursue this direction [173,204], albeit without explicitly grounding it in metacognition; we suggest that a metacognitive lens offers new research and design directions for augmenting GenAI system usability.”

• cites [173] 173. Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023. Sensecape: En- abling Multilevel Exploration and Sensemaking with Large Language Models. https://doi.org/10.48550/arXiv.2305.11483 arXiv:2305.11483 [cs]. sourcePartly supported

  Section 3.2 describes helping users understand a model’s limitations and decide how to incorporate its output into further actions, which is related to the claimed direction. But the paper addresses interpretable machine learning, not integrating metacognitive support into GenAI systems; it does not substantiate that more specific attribution. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [204] 204. Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Trans- parent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-22. https://doi.org/10.1145/3491102.3517582 sourceCould not verify

  The Abstract and Conclusion discuss human confidence judgments and error monitoring, not HCI or GenAI systems. Searches for HCI, generative AI, system design, and usability found no passage addressing whether recent HCI work integrates metacognitive support. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

People often implicitly and unintentionally rely on heuristics to guide their metacognitive monitoring and control.

“People often implicitly and unintentionally rely on heuristics to guide their metacognitive monitoring and control [3].”

• cites [3] 3. Rakefet Ackerman. 2019. Heuristic Cues for Meta-Reasoning Judgments: Review and Methodology. Psihologijske teme 28, 1 (May 2019), 1-20. https://doi.org/10. 31820/pt.28.1.1 Publisher: Filozofski fakultet u Rijeci. sourcePartly supported

  The Introduction says metacognitive judgments rely on heuristic cues, are assumed to occur spontaneously, and guide control over effort allocation. It does not establish that reliance on the cues is implicit or unintentional; spontaneity of the judgments alone does not settle that part of the claim. No page numbers are available in our copy. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Confidence and its adjustment are central to decision-making and reasoning, especially in many aspects of human-AI interaction.

“Confidence and its adjustment are central to decision-making and reasoning, especially in many aspects of human-AI interaction [4, 171] ( §3.1, §3.2, and §3.3).”

• cites [4] 4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1-15. sourceNot checked — no full text of the source
• cites [171] 171. Mark Steyvers and Aakriti Kumar. 2023. Three Challenges for AI-Assisted Decision-Making. Perspectives on Psychological Science: A Journal of the As- sociation for Psychological Science (July 2023), 17456916231181102. https: //doi.org/10.1177/17456916231181102 sourceSupported

HCI has so far considered metacognition mainly in the context of computer science education.

“Nevertheless, the field of humancomputer interaction (HCI) has so far considered metacognition mainly in the context of computer science education [107,142].”

• cites [107] 107. Dastyni Loksa, Lauren Margulieux, Brett A. Becker, Michelle Craig, Paul Denny, Raymond Pettit, and James Prather. 2022. Metacognition and Self-Regulation in Programming Education: Theories and Exemplars of Use. ACM Transactions on Computing Education 22, 4 (Sept. 2022), 39:1-39:31. https://doi.org/10.1145/ 3487050 sourceNot checked — no full text of the source
• cites [142] 142. James Prather, Brett A. Becker, Michelle Craig, Paul Denny, Dastyni Loksa, and Lauren Margulieux. 2020. What Do We Think We Think We Are Doing?: Metacognition and Self-Regulation in Programming. In Proceedings of the 2020 ACM Conference on International Computing Education Research. ACM, Virtual Event New Zealand, 2-13. https://doi.org/10.1145/3372782.3406263 sourceCould not verify

  Section 2 discusses metacognition in relation to mind wandering and creativity, while Section 4 discusses its development in school. Searches for HCI, human-computer interaction, and computer science education found no passage assessing where HCI has mainly considered metacognition. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Metacognitive interventions, such as training through feedback on metacognitive performance, can increase judgment accuracy.

“Metacognitive interventions, such as training through feedback on metacognitive performance [30], can, for example, increase judgment accuracy-the ability to distinguish between one's own correct and incorrect metacognitive processing.”

• cites [30] 30. Jason Carpenter, Maxine T. Sherman, Rogier A. Kievit, Anil K. Seth, Hakwan Lau, and Stephen M. Fleming. 2019. Domain-General Enhancements of Metacognitive Ability Through Adaptive Training. Journal of Experimental Psychology. General 148, 1 (Jan. 2019), 51-64. https://doi.org/10.1037/xge0000505 sourceSupported

Programmers who were highly confident in their own ability actively questioned the AI code assistant when it produced confusing output.

“By contrast, programmers who were highly confident in their own ability actively questioned the AI code assistant when it produced confusing output [198].”

• cites [198] 198. Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces (IUI '21). Association for Computing Machinery, New York, NY, USA, 402-412. https://doi.org/10.1145/ 3397481.3450656 sourceSupported

Model flexibility in GenAI systems presents a 'fuzzy abstraction matching' problem-it becomes difficult for users to discern a system's capabilities and to match one's intent and prompting accordingly.

“which presents what Sarkar et al. [156] term a 'fuzzy abstraction matching' problem-it becomes difficult for users to discern a system's capabilities and to match one's intent and prompting accordingly”

• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  The Abstract and Discussion address uncertainty-guided information search and dogmatism in a perceptual decision task; they do not address GenAI capabilities or prompting. Searches for “fuzzy abstraction matching” and for GenAI model flexibility found no passage supporting the attributed claim. The cited paper’s authors also do not match the Sarkar et al. named in the citing sentence. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

Non-determinism of GenAI exacerbates confidence adjustment challenges, particularly when tweaking one aspect of the prompt might unintentionally change a different aspect of the output.

“This is exacerbated by the non-determinism of GenAI, particularly when tweaking one aspect of the prompt might unintentionally change a different aspect of the output [33].”

• cites [33] 33. Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. 2023. Machine Expla- nations and Human Understanding. https://doi.org/10.48550/arXiv.2202.04092 arXiv:2202.04092 [cs]. sourceCould not verify

  The Exit survey passage says the framework can treat model decision boundaries and errors probabilistically, but it does not discuss GenAI or prompt changes. The passages on prediction shown and trust address human understanding of model error, not whether changing one part of a prompt unintentionally changes another part of the output. Searches for prompt non-determinism and confidence adjustment found no passage addressing that claim. (LLM, conf=0.75)

  Experimental — accuracy not yet measured

Evaluating these outputs for quality therefore becomes far more important and effortful (in terms of cognitive load) compared to 'auto-complete' phrase suggestions or intelligent code completion.

“Evaluating these outputs for quality therefore becomes far more important and effortful (in terms of cognitive load) compared to 'auto-complete' phrase suggestions or intelligent code completion [156,157].”

• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  The Results (Measuring Information Search) examine decisions about whether to seek additional evidence in a dot-judgment task. The Discussion considers uncertainty and effort in metacognitive control, but does not compare the cognitive load of evaluating generated outputs with autocomplete or code completion. Searches for those terms and that comparison found no supporting passage in our copy; page numbers were unavailable. (LLM, conf=0.74)

  Experimental — accuracy not yet measured
• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourcePartly supported

  The paper treats validating whether an output is fit for purpose as a cost (Table 2, System) and says users’ cognitive effort matters when evaluating multimodal systems (§2, “A New Kind of Cognitive Tool”). It does not establish that assessing their outputs is more important or more cognitively effortful than assessing auto-complete suggestions or intelligent code completion. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Programmers were reluctant to deeply review and repair AI-generated code, preferring instead to re-write the entire code themselves.

“Programmers were reluctant to deeply review and repair AI-generated code, preferring instead to re-write the entire code themselves [188].”

• cites [188] 188. Priyan Vaithilingam, Tianyi Zhang, and Elena L. Glassman. 2022. Expectation vs. Experience: Evaluating the Usability of Code Generation Tools Powered by Large Language Models. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA '22). Association for Computing Machinery, New York, NY, USA, 1-7. https://doi.org/10.1145/3491101.3519665 sourcePartly supported

  In “Granularity of uncertainty highlighting” (page unavailable), one participant said they would probably discard and rewrite a completion if its score were low. That conditional remark does not establish that programmers generally preferred rewriting over reviewing and repairing generated code; Section 5.2 instead reports participants making targeted edits. (LLM, conf=0.86)

  Experimental — accuracy not yet measured

GenAI can produce (often extensive) content, such as entire emails, presentations, or software.

“GenAI can produce (often extensive) content, such as entire emails, presentations, or software.”

• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  The Materials and Methods passages describe an online dot-discrimination task, while the References concern information seeking and related research. Searches for GenAI and for generating emails, presentations, or software found no passage in our copy addressing those capabilities. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourcePartly supported

  The Introduction describes generative models producing flexible outputs, and §2, “A New Kind of Cognitive Tool,” says they can generate novel content. The modalities table includes code generation. Our copy does not establish the more specific claim that they produce entire emails, presentations, or software; §4 instead gives an auto-completed piece of code as an example. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Manufacturing designers who successfully learned to co-create with GenAI abstracted and explained the problem to themselves.

“Beyond coding, manufacturing designers who successfully learned to co-create with GenAI abstracted and explained the problem to themselves [70]”

• cites [70] 70. Frederic Gmeiner, Humphrey Yang, Lining Yao, Kenneth Holstein, and Nikolas Martelaro. 2023. Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools. In Pro- ceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, 1-20. https://doi.org/10.1145/3544548.3580999 sourceSupported

These difficulties were exacerbated by the many parameters available to users who grappled with understanding and using them effectively.

“These difficulties were exacerbated by the many parameters available to users who grappled with understanding and using them effectively [70,83,204].”

• cites [70] 70. Frederic Gmeiner, Humphrey Yang, Lining Yao, Kenneth Holstein, and Nikolas Martelaro. 2023. Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools. In Pro- ceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, 1-20. https://doi.org/10.1145/3544548.3580999 sourceSupported
• cites [83] 83. Dhanya Jayagopal, Justin Lubin, and Sarah E. Chasins. 2022. Exploring the Learnability of Program Synthesizers by Novice Programmers. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22). Association for Computing Machinery, New York, NY, USA, 1-15. https://doi.org/10.1145/3526113.3545659 sourceCould not verify

  Section 3.2 discusses parameters as knowledge stored within a model, while section 5.2.2 discusses controls for generation. Neither addresses whether users faced many available parameters or struggled to understand and use them effectively. Searches for user-configurable settings and difficulty using them found no passage supporting the claim. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [204] 204. Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Trans- parent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-22. https://doi.org/10.1145/3491102.3517582 sourceCould not verify

  The Shared limitations section discusses whether models of confidence and error monitoring apply to real-world decisions, not whether users struggle with numerous available parameters. Searches for users’ difficulty understanding and using parameters found no passage addressing that claim; the paper’s discussion of model parameters does not establish it. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Processing fluency-the subjective ease with which a cognitive task is performed-can influence people's confidence in information accuracy.

“One relevant type of cue-'processing fluency',”

• cites [3] 3. Rakefet Ackerman. 2019. Heuristic Cues for Meta-Reasoning Judgments: Review and Methodology. Psihologijske teme 28, 1 (May 2019), 1-20. https://doi.org/10. 31820/pt.28.1.1 Publisher: Filozofski fakultet u Rijeci. sourceSupported
• cites [201] 201. Jennifer Wiley, Thomas D. Griffin, Allison J. Jaeger, Andrew F. Jarosz, Patrick J. Cushen, and Keith W. Thiede. 2016. Improving metacomprehension accuracy in an undergraduate course context. Journal of Experimental Psychology: Applied 22, 4 (Dec. 2016), 393-405. https://doi.org/10.1037/xap0000096 sourceCould not verify

  The Introduction (p. 2) and Study Results (p. 6) discuss LLM prompt chaining, not processing fluency or judgments of information accuracy. Searches for “processing fluency” and for ease of processing affecting confidence found no passage addressing the claim in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

The generative nature of GenAI means that many user workflows will or have shifted from users generating content to evaluating it.

“The generative nature of GenAI-its 'originality' [157]-means that many user workflows will or have shifted from users generating content to evaluating it [149,154], as already documented in programming [156] and manufacturing design [70].”

• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourcePartly supported

  Section 4, “Elements of the Prompting Process” (page unavailable), describes generated images and code as potential outputs that save users from producing them themselves. Its “System” table also identifies user validation of an answer as a step. These passages support a generate-and-evaluate workflow, but do not establish that many user workflows have shifted, or will shift, from content creation to evaluation. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [149] 149. Evan F. Risko and Megan O. Kelly. 2023. Thinking in the digital age: Everyday cognition and the dawn of a new age of metacognition research. Applied Cognitive Psychology 37, 4 (2023), 785-788. https://doi.org/10.1002/acp.4102 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.4102. sourceCould not verify

  The Abstract and 'How does confidence guide behavior?' section discuss evaluating one’s own cognitive performance, not evaluating GenAI-generated content. Searches for GenAI, content creation, and workflow changes found no passage addressing a shift from generating content to evaluating it. Page numbers were unavailable in our copy. (LLM, conf=0.76)

  Experimental — accuracy not yet measured
• cites [154] 154. Advait Sarkar. 2023. Exploring Perspectives on the Impact of Artificial In- telligence on the Creativity of Knowledge Work: Beyond Mechanised Pla- giarism and Stochastic Parrots. In Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work (CHIWORK '23). Association for Computing Machinery, New York, NY, USA, 1-17. https: //doi.org/10.1145/3596671.3597650 sourceSupported
• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourceCould not verify

  The Discussion describes how dogmatism relates to information search under uncertainty, not how GenAI changes user workflows. Searches for GenAI, content generation, and a shift toward evaluating generated content found no passage addressing the attributed claim. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Simple textual prompts promoting self-awareness significantly decreased participants' susceptibility to incorrect (albeit realistic) information.

“For example, simple textual prompts promoting self-awareness significantly decreased participants' susceptibility to incorrect (albeit realistic) information [153].”

• cites [153] 153. Nikita A. Salovich and David N. Rapp. 2021. Misinformed and unaware? Metacognition and the influence of inaccurate information. Journal of Ex- perimental Psychology: Learning, Memory, and Cognition 47, 4 (2021), 608-624. https://doi.org/10.1037/xlm0000977 Place: US Publisher: American Psychologi- cal Association. sourceCould not verify

  The section “Towards a Doctrine of Negotiated Complexity” discusses simple in-app prompts for teaching software features, but not prompts promoting self-awareness or a measured reduction in susceptibility to incorrect information. Searches for those concepts and for misinformation interventions found no passage in our copy reporting the claimed result. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

People can engage in multi-level planning, where hours, days, and weeks have to be considered simultaneously.

“For example, people can engage in multi-level planning, where hours, days, and weeks have to be considered simultaneously [6].”

• cites [6] 6. Yoana Ahmetoglu, Duncan P. Brumby, and Anna L. Cox. 2021. To Plan or Not to Plan? A Mixed-Methods Diary Study Examining When, How and Why Knowledge Work Planning is Inaccurate. Proceedings of the ACM on Human- Computer Interaction 4, CSCW3 (Jan. 2021), 222:1-222:20. https://doi.org/10. 1145/3432921 sourceSupported

GenAI models are non-deterministic.

“Moreover, as noted, GenAI models are non-deterministic [136].”

• cites [136] 136. Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. 2023. LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation. http://arxiv.org/abs/2308.02828 arXiv:2308.02828 [cs]. sourceCould not verify

  The Introduction discusses imperfect machine-learning algorithms, and the Procedure describes a machine-vision algorithm used to estimate jelly-bean counts. Neither passage addresses generative AI or whether models produce different outputs for the same input; searches for non-determinism, randomness, and repeated-input behavior found no supporting passage in our copy. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

This requires developing an expertise and a well-adjusted confidence that is distinct from existing domain expertise.

“As it stands, this requires developing an expertise and a well-adjusted confidence that is distinct from existing domain expertise, with, for example,”

• cites [156] 156. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. sourcePartly supported

  The Discussion describes confidence-guided information search as metacognitive control and suggests that training could target it. It does not establish that people must develop a separate expertise or well-adjusted confidence distinct from their existing domain expertise; the training proposal is tentative. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Research on real-world GenAI workflows, primarily in AI-assisted coding, suggests that tools like GitHub Copilot alter users' workflows in diverse ways.

“Research on real-world GenAI workflows, primarily in AI-assisted coding, suggests that tools like GitHub Copilot alter users' workflows in diverse ways [166].”

• cites [166] 166. Auste Simkute, Lev Tankelevitch, Viktor Kewenig, Ava Elizabeth Scott, Abigail Sellen, and Sean Rintel. 2024. Ironies of Generative AI: Understanding and mitigating productivity loss in human-AI interactions. https://doi.org/10. 48550/arXiv.2402.11364 arXiv:2402.11364 [cs]. sourceSupported

Self-evaluation is particularly promising for augmenting GenAI explainability, where it can help increase users' receptiveness to explanations that seek to update their mental models of GenAI systems.

“Self-evaluation is particularly promising for augmenting GenAI explainability, where it can help increase users' receptiveness to explanations that seek to update their mental models of GenAI systems [4,171] (see §4.2.1 for more on explainability).”

• cites [4] 4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1-15. sourceNot checked — no full text of the source
• cites [171] 171. Mark Steyvers and Aakriti Kumar. 2023. Three Challenges for AI-Assisted Decision-Making. Perspectives on Psychological Science: A Journal of the As- sociation for Psychological Science (July 2023), 17456916231181102. https: //doi.org/10.1177/17456916231181102 sourcePartly supported

  In 'Improving the Assessment of Mental Models' (page unavailable), the paper suggests using metacognition to understand how people assess their own confidence. In 'What information should the AI present to the human DM?' (page unavailable), it suggests that interventions increasing cognitive effort might help people use explanations effectively. It does not say that self-evaluation increases receptiveness to explanations or helps explanations update users’ mental models of GenAI. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

This imposes a distinct metacognitive demand on users that must make these decisions.

“We argue that this imposes a distinct metacognitive demand on users that must make these decisions, akin to [149], who make a similar argument for digital storage and memory.”

• cites [149] 149. Evan F. Risko and Megan O. Kelly. 2023. Thinking in the digital age: Everyday cognition and the dawn of a new age of metacognition research. Applied Cognitive Psychology 37, 4 (2023), 785-788. https://doi.org/10.1002/acp.4102 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.4102. sourceCould not verify

  The Abstract defines metacognition as monitoring and controlling cognitive processes, including memory and decisions, but does not discuss digital storage. Searches for digital storage and decisions about what to store found no passage making the similar argument attributed to this source. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

Similar systems have been shown to improve users' planning and other metacognitive processes.

“Similar systems have been shown to improve users' planning and other metacognitive processes [38].”

• cites [38] 38. Anita Crescenzi, Austin R. Ward, Yuan Li, and Rob Capra. 2021. Supporting Metacognition during Exploratory Search with the OrgBox. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '21). Association for Computing Machinery, New York, NY, USA, 1197-1207. https://doi.org/10.1145/3404835.3462955 sourcePartly supported

  Section 5.4 reports that OrgBox users perceived significantly greater support for monitoring and evaluation than users of the comparison tool. Planning support was rated higher, but the difference was not statistically significant (p=.059). The study measured perceived support, not demonstrated improvements in users’ metacognitive processes. (LLM, conf=0.88)

  Experimental — accuracy not yet measured

GenAI tools can have multiple and often non-intuitive failure modes.

“Users' confidence and their ability to adjust it may also be challenged by the fact that GenAI tools can have multiple and often non-intuitive failure modes [33,157].”

• cites [33] 33. Chacha Chen, Shi Feng, Amit Sharma, and Chenhao Tan. 2023. Machine Expla- nations and Human Understanding. https://doi.org/10.48550/arXiv.2202.04092 arXiv:2202.04092 [cs]. sourcePartly supported

  Section 5.2 discusses model errors arising from invalid input–output relationships, including complex nonlinear ones. The section on explanations without task-specific intuitions says users may be unable to identify model errors. These passages support difficulty recognizing AI errors, but they do not establish that GenAI tools have multiple, often non-intuitive failure modes. (LLM, conf=0.78)

  Experimental — accuracy not yet measured
• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourceSupported

Feedforward information can increase cognitive load.

“However, feedforward information can also increase cognitive load [42], so the right balance, particularly for complex systems, remains to be explored.”

• cites [42] 42. Hai Dang, Lukas Mecke, and Daniel Buschek. 2022. GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-15. https://doi.org/10.1145/3491102.3502141 sourceSupported

The quality of generated content and its uses may be more difficult to evaluate objectively.

“but the quality of generated content and its uses may be more difficult to evaluate objectively (e.g., consider how one would objectively evaluate the quality of an LLM-generated email) [87,99].”

• cites [87] 87. Sucharit Katyal and Stephen Fleming. 2023. Construct validity in metacognition research: balancing the tightrope between rigor of measurement and breadth of construct. https://doi.org/10.31234/osf.io/etjqh sourcePartly supported

  The Affective metacognition section (page unavailable) says that some subjective evaluations lack an objective ground truth, supporting the general difficulty of objective assessment. It does not address the quality or uses of generated content, including LLM-generated emails. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [99] 99. Vivian Lai, Chacha Chen, Alison Smith-Renner, Q. Vera Liao, and Chenhao Tan. 2023. Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency (FAccT '23). Association for Computing Machinery, New York, NY, USA, 1369-1385. https: //doi.org/10.1145/3593013.3594087 sourcePartly supported

  Section 4.2.2 (p. 11) says there are few metrics for determining the quality of generated explanations, making it hard to decide when to disregard them. This supports the difficulty of evaluating explanations and their use, but the paper does not extend that point to generated content generally or to LLM-generated emails. (LLM, conf=0.84)

  Experimental — accuracy not yet measured

Self-explanation prompts improved students' accuracy in evaluating their own understanding of information.

“Likewise, self-explanation prompts improved students' accuracy in evaluating their own understanding of information [201].”

• cites [201] 201. Jennifer Wiley, Thomas D. Griffin, Allison J. Jaeger, Andrew F. Jarosz, Patrick J. Cushen, and Keith W. Thiede. 2016. Improving metacomprehension accuracy in an undergraduate course context. Journal of Experimental Psychology: Applied 22, 4 (Dec. 2016), 393-405. https://doi.org/10.1037/xap0000096 sourceCould not verify

  The Study Design (p. 5) describes four designers and developers building LLM prompt chains, not students responding to self-explanation prompts. The Study Results (pp. 5–6) discuss chain construction and debugging, but do not assess students’ accuracy in judging their own understanding. Searches for self-explanation and metacognitive accuracy found no relevant passage. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

The generality of GenAI-its applicability to a wide range of tasks-poses a higher-level question to end-users about their workflow automation strategy.

“Beyond the metacognitive demands implied in the local interaction with GenAI, the generality of GenAI-its applicability to a wide range of tasks [157]-poses a higher-level question to end-users about their workflow automation strategy”

• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourceSupported

Interventions that cue users to reflect on their goals and strategies via question prompts or conversational interfaces have been shown to improve outcomes in the workplace, education, and other domains.

“Interventions that cue users to reflect on their goals and strategies via question prompts or conversational interfaces have been shown to improve outcomes in the workplace [92,122,209], education [51], and other domains [17,147].”

• cites [92] 92. Rafal Kocielnik, Lillian Xiao, Daniel Avrahami, and Gary Hsieh. 2018. Reflection Companion: A Conversational System for Engaging Users in Reflection on Physical Activity. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 2 (July 2018), 1-26. https://doi.org/10.1145/3214273 sourceNot checked — no full text of the source
• cites [122] 122. André N. Meyer, Gail C. Murphy, Thomas Zimmermann, and Thomas Fritz. 2021. Enabling Good Work Habits in Software Developers through Reflective Goal- Setting. IEEE Transactions on Software Engineering 47, 9 (Sept. 2021), 1872-1885. https://doi.org/10.1109/TSE.2019.2938525 Conference Name: IEEE Transactions on Software Engineering. sourcePartly supported

  Section 5.3.2 (Dialogue; page unavailable) describes explanatory dialogue systems that use probing questions, including in intelligent tutoring. It does not show that prompts cue users to reflect on their goals and strategies or that these systems improve outcomes; it says the evaluation of EDGE was anecdotal, with no formal comparison. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [209] 209. Nima Zargham, Leon Reicherts, Michael Bonfert, Sarah Theres Voelkel, Johannes Schoening, Rainer Malaka, and Yvonne Rogers. 2022. Understanding Circum- stances for Desirable Proactive Behaviour of Voice Assistants: The Proactivity Dilemma. In Proceedings of the 4th Conference on Conversational User Interfaces (CUI '22). Association for Computing Machinery, New York, NY, USA, 1-14. https://doi.org/10.1145/3543829.3543834 sourcePartly supported

  The Conclusion (p. not available) likens code completion to a chatbot conversation and links Copilot usage to developers’ perceived productivity. It does not study prompts or interfaces that cue reflection on goals and strategies, or show that such interventions improve workplace outcomes. (LLM, conf=0.80)

  Experimental — accuracy not yet measured
• cites [51] 51. Anneline Devolder, Johan van Braak, and Jo Tondeur. 2012. Supporting self- regulated learning in computer-based learning environments: Systematic review of effects of scaffolding in the domain of science education. Journal of Computer Assisted Learning 28 (Dec. 2012). https://doi.org/10.1111/j.1365-2729.2011.00476. x sourceNot checked — no full text of the source
• cites [17] 17. Marit Bentvelzen, Paweł W. Woźniak, Pia S.F. Herbes, Evropi Stefanidi, and Jasmin Niess. 2022. Revisiting Reflection in HCI: Four Design Resources for Technologies that Support Reflection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 1 (March 2022), 2:1-2:27. https: //doi.org/10.1145/3517233 sourcePartly supported

  Section 5.2.2 describes conversational systems that ask questions to stimulate reflection, including a workplace chatbot. The Introduction also describes reflection-supporting systems across diverse settings. But the paper does not establish that prompts about goals and strategies improve outcomes in the workplace, education, and other domains; the Introduction says which techniques are likely to be effective remains unclear. (LLM, conf=0.85)

  Experimental — accuracy not yet measured
• cites [147] 147. Leon Reicherts and Yvonne Rogers. 2020. Do Make me Think! How CUIs Can Support Cognitive Processes. In Proceedings of the 2nd Conference on Conver- sational User Interfaces (CUI '20). Association for Computing Machinery, New York, NY, USA, 1-4. https://doi.org/10.1145/3405755.3406157 sourceCould not verify

  The Introduction describes questions about people’s goals for their photographs, but not an intervention designed to improve outcomes. Section 3 discusses monitoring and control when using digital tools; neither it nor the passages found by searching for question prompts, conversational interfaces, and interventions reports improved outcomes from reflective prompts in any domain. (LLM, conf=0.74)

  Experimental — accuracy not yet measured

System customizability can moderate the metacognitive demands of GenAI.

“The question of how much control to give users-system customizability, or how many 'knobs' a user can adjust-is another design choice that can moderate the metacognitive demands of GenAI [42,119].”

• cites [42] 42. Hai Dang, Lukas Mecke, and Daniel Buschek. 2022. GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Information. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-15. https://doi.org/10.1145/3491102.3502141 sourcePartly supported

  Section 5.10 (pp. 8–9) reports that more sliders increased perceived workload and made decisions about which slider to adjust next harder, supporting the point that the number of controls affects user demands. The study tested an image-generation task, however, and did not directly measure metacognitive demands across GenAI systems. (LLM, conf=0.83)

  Experimental — accuracy not yet measured
• cites [119] 119. Andrew M Mcnutt, Chenglong Wang, Robert A Deline, and Steven M. Drucker. 2023. On the Design of AI-powered Code Assistants for Notebooks. In Pro- ceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI '23). Association for Computing Machinery, New York, NY, USA, 1-16. https://doi.org/10.1145/3544548.3580940 sourceNot checked — no full text of the source

One could be well-calibrated on average but still have poor resolution.

“One could be wellcalibrated on average (e.g., one's overall level of confidence matches one's overall level of accuracy) but still have poor resolution (i.e., one's confidence level does not vary much between correct and incorrect answers)[66].”

• cites [66] 66. Stephen M. Fleming and Hakwan C. Lau. 2014. How to measure metacognition. Frontiers in Human Neuroscience 8 (2014). https://www.frontiersin.org/articles/ 10.3389/fnhum.2014.00443 sourceSupported

As we offload more of our cognition to today's GenAI systems, the demand for our metacognition increases.

“Analogously, as we offload more of our cognition to today's GenAI systems, the demand for our metacognition increases [149].”

• cites [149] 149. Evan F. Risko and Megan O. Kelly. 2023. Thinking in the digital age: Everyday cognition and the dawn of a new age of metacognition research. Applied Cognitive Psychology 37, 4 (2023), 785-788. https://doi.org/10.1002/acp.4102 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/acp.4102. sourceCould not verify

  The Abstract defines metacognition as evaluating and controlling one’s own cognitive processes, and “How does confidence guide behavior?” discusses its role in learning and cognitive control. Neither passage addresses whether offloading cognition to GenAI increases the demand for metacognition. Searches for cognitive offloading, external tools, and increased metacognitive demand found no passage making that connection. (LLM, conf=0.76)

  Experimental — accuracy not yet measured

GenAI systems are relevantly distinct from search engines.

“Although the perspective of metacognition is equally relevant for understanding the usability of search engines and similar technologies, we focus our scope to GenAI systems as they are relevantly distinct from that of search engines[157].”

• cites [157] 157. Wout Schellaert, Fernando Martínez-Plumed, Karina Vold, John Burden, Pablo A. M. Casares, Bao Sheng Loe, Roi Reichart, Sean Ó hÉigeartaigh, Anna Korho- nen, and José Hernández-Orallo. 2023. Your Prompt is My Command: On Assess- ing the Human-Centred Generality of Multimodal Models. Journal of Artificial Intelligence Research 77 (June 2023), 377-394. https://doi.org/10.1613/jair.1.14157 sourceSupported

The two aspects of confidence can be independent.

“The two aspects of confidence can be independent.”

• cites [66] 66. Stephen M. Fleming and Hakwan C. Lau. 2014. How to measure metacognition. Frontiers in Human Neuroscience 8 (2014). https://www.frontiersin.org/articles/ 10.3389/fnhum.2014.00443 sourceSupported

The design of intuitive interfaces for GenAI systems might inadvertently give users a misleading sense of competence, increasing the risk of errors.

“The design of intuitive interfaces for GenAI systems might therefore inadvertently give users a misleading sense of competence, increasing the risk of errors (likewise discussed in cognitive psychology [153]).”

• cites [153] 153. Nikita A. Salovich and David N. Rapp. 2021. Misinformed and unaware? Metacognition and the influence of inaccurate information. Journal of Ex- perimental Psychology: Learning, Memory, and Cognition 47, 4 (2021), 608-624. https://doi.org/10.1037/xlm0000977 Place: US Publisher: American Psychologi- cal Association. sourcePartly supported

  Section 2 (‘How Did We Get Here?’, page unavailable) warns that software can be made to appear more usable while becoming less useful, supporting the general concern about intuitive design. The paper does not establish that GenAI interfaces give users a misleading sense of their own competence or increase errors. (LLM, conf=0.78)

  Experimental — accuracy not yet measured

There is a risk that strategies to address metacognitive demands may also increase cognitive load, due to the additional information that users have to process.

“There is a risk that strategies to address metacognitive demands may also increase cognitive load, due to the additional information that users have to process, such as self-reflective prompts, a set of sub-goals resulting from task decomposition, or model explanations [46].”

• cites [46] 46. Anique B. H. de Bruin, Julian Roelle, Shana K. Carpenter, Martine Baars, and EFG-MRE. 2020. Synthesizing Cognitive Load and Self-regulation Theory: a Theoretical Framework and Research Agenda. Educational Psychology Review 32, 4 (Dec. 2020), 903-915. https://doi.org/10.1007/s10648-020-09576-4 sourceSupported

Having well-calibrated confidence does not necessarily imply high confidence resolution.

“Having a well-calibrated confidence does not necessarily imply high confidence resolution.”

• cites [66] 66. Stephen M. Fleming and Hakwan C. Lau. 2014. How to measure metacognition. Frontiers in Human Neuroscience 8 (2014). https://www.frontiersin.org/articles/ 10.3389/fnhum.2014.00443 sourceSupported

Indicating model uncertainty and an explanation for that uncertainty can help users disentangle the role of their prompting and output evaluation ability from that of models' capabilities, further supporting confidence adjustment.

“Finally, indicating model uncertainty [19,174,198], for example, by means of color-coding [167], and an explanation for that uncertainty [5,190], can also help users disentangle the role of their prompting and output evaluation ability from that of models' capabilities, further supporting confidence adjustment [197].”

• cites [19] 19. Umang Bhatt, Javier Antorán, Yunfeng Zhang, Q. Vera Liao, Prasanna Sattigeri, Riccardo Fogliato, Gabrielle Melançon, Ranganath Krishnan, Jason Stanley, Omesh Tickoo, Lama Nachman, Rumi Chunara, Madhulika Srikumar, Adrian Weller, and Alice Xiang. 2021. Uncertainty as a Form of Transparency: Mea- suring, Communicating, and Using Uncertainty. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (AIES '21). Association for Computing Machinery, New York, NY, USA, 401-413. https://doi.org/10.1145/ 3461702.3462571 sourceSupported
• cites [174] 174. Jiao Sun, Q. Vera Liao, Michael Muller, Mayank Agarwal, Stephanie Houde, Kartik Talamadupula, and Justin D. Weisz. 2022. Investigating Explainability of Generative AI for Code through Scenario-based Design. In 27th International Conference on Intelligent User Interfaces (IUI '22). Association for Computing Machinery, New York, NY, USA, 212-228. https://doi.org/10.1145/3490099. 3511119 sourceNot checked — no full text of the source
• cites [198] 198. Justin D. Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I. Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces (IUI '21). Association for Computing Machinery, New York, NY, USA, 402-412. https://doi.org/10.1145/ 3397481.3450656 sourceSupported
• cites [167] 167. Sofia Eleni Spatharioti, David M. Rothschild, Daniel G. Goldstein, and Jake M. Hofman. 2023. Comparing Traditional and LLM-based Search for Con- sumer Choice: A Randomized Experiment. http://arxiv.org/abs/2307.03744 arXiv:2307.03744 [cs]. sourceNot checked — no full text of the source
• cites [5] 5. Mayank Agarwal, Kartik Talamadupula, Stephanie Houde, Fernando Martinez, Michael Muller, John Richards, Steven Ross, and Justin D. Weisz. 2021. Quality Estimation & Interpretability for Code Translation. https://doi.org/10.48550/ arXiv.2012.07581 arXiv:2012.07581 [cs]. sourcePartly supported

  Section 2.1.2 (p. 2) reports that users struggled to understand when and why the code-translation model lacked confidence, and Section 4.2 (p. 4) identifies a need for interpretable confidence values. This supports the cited point about explaining uncertainty, but the paper does not show that such explanations help users separate their prompting or output-evaluation ability from model capability. (LLM, conf=0.82)

  Experimental — accuracy not yet measured
• cites [190] 190. Helena Vasconcelos, Gagan Bansal, Adam Fourney, Q. Vera Liao, and Jen- nifer Wortman Vaughan. 2023. Generation Probabilities Are Not Enough: Exploring the Effectiveness of Uncertainty Highlighting in AI-Powered Code Completions. https://doi.org/10.48550/arXiv.2302.07248 arXiv:2302.07248 [cs]. sourceNot checked — no full text of the source

GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control.

“Drawing on research in psychology and cognitive science, and recent GenAI user studies, we illustrate how GenAI systems impose metacognitive demands on users, requiring a high degree of metacognitive monitoring and control.”

These metacognitive demands are induced by GenAI's model flexibility, generality, and originality.

“We posit that these metacognitive demands are induced by GenAI's model flexibility, generality, and originality.1”

Many GenAI systems require that a task be broken up into sub-tasks.

“Moreover, it often requires that a task be broken up into sub-tasks (”

The demand for self-awareness and task decomposition is exacerbated by model flexibility in GenAI: many parameters for end-users to adjust.

“This demand for self-awareness and task decomposition is exacerbated by a particular type of model flexibility in GenAI: today's systems afford many parameters for end-users to adjust; these can be formal parameters like the model temperature, or a range of unspecified parameters that can be adjusted through text prompting (e.g., the tone, level of detail, or structure of a piece of text).”

Prompting in current GenAI systems imposes a high metacognitive demand due to the need for self-awareness of goals, increasing cognitive load.

“For example, as we illustrate below, prompting in current GenAI systems imposes a high metacognitive demand due to the need for self-awareness of goals, increasing cognitive load, while the interaction method of typing (rather than speaking) further increases cognitive load, albeit without much associated metacognitive demand.”

Metacognition offers a valuable and unexplored perspective to understand and design for the usability challenges of GenAI.

“Here, we argue that metacognition-the psychological ability to monitor and control one's own thought processes [4,62,131,181]-offers a valuable and unexplored perspective to understand and design for the usability challenges of GenAI.”

• AssumptionRests on an assumption: There is not yet a coherent understanding of the usability challenges of GenAI, much less one grounded in a theory of human cognition.

Many GenAI systems require specification of implicit goals (e.g., that an email is to a senior colleague and needs an appropriate tone).

“Many GenAI systems require specification that the email is to a senior colleague and needs an appropriate tone.”

Iterative output evaluation and adjustment (prompt iteration) depends on users' confidence in their prompting ability, and metacognitive flexibility to adapt their prompting strategy.

“Next, iterative output evaluation and adjustment (prompt iteration) depends on users' confidence in their prompting ability, and metacognitive flexibility to adapt their prompting strategy (Figure 2b).”

This model flexibility and control afforded to users requires knowing what one wants to achieve and conveying that explicitly and effectively to the system.

“This model flexibility and control afforded to users requires knowing what one wants to achieve and convey that explicitly and effectively to the system.”

Monitoring and control are interrelated.

“Monitoring and control are interrelated [93,95].”

• cites [93] 93. Asher Koriat. 2007. Metacognition and consciousness. In The Cambridge handbook of consciousness. Cambridge University Press, New York, NY, US, 289-325. https://doi.org/10.1017/CBO9780511816789.012 sourceNot checked — no full text of the source
• cites [95] 95. Asher Koriat, Hilit Ma'ayan, and Ravit Nussinson. 2006. The intricate rela- tionships between monitoring and control in metacognition: Lessons for the cause-and-effect relation between subjective experience and behavior. Jour- nal of Experimental Psychology: General 135, 1 (Feb. 2006), 36-69. https: //doi.org/10.1037/0096-3445.135.1.36 sourceNot checked — no full text of the source

The ability of GenAI systems to quickly and easily generate extensive content may serve as a cue that misleadingly increases users' confidence, not only in the output itself, but also in their own ability to evaluate it.

“Analogously, the ability of GenAI systems to quickly and easily generate extensive content may serve as a cue that misleadingly increases users' confidence, not only in the output itself, but also in their own ability to evaluate it.”

The challenge of output evaluation is present even in interaction modes without user prompting, such as in GitHub Copilot.

“The challenge of output evaluation is present even in interaction modes without user prompting, such as in GitHub Copilot, which includes automated suggestions.”

Systems that provide automated suggestions such as GitHub Copilot alleviate some of the demands associated with prompting.

“Systems that provide automated suggestions such as GitHub Copilot alleviate some of the demands associated with prompting.”

In [208], non-expert participants using an LLM-based tool to improve a chatbot experienced a struggle getting started.

“In [208], non-expert participants used an LLM-based tool to improve a chatbot through prompting.One of the challenges they experienced was a struggle getting started.7”

• cites [208] 208. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why Johnny Can't Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. ACM, Hamburg Germany, 1-21. https://doi.org/10.1145/ 3544548.3581388 sourceNot checked — no full text of the source

These challenges partly stem from a lack of feedback in the system about prompting effectiveness.

“To be clear, these challenges partly stem from a lack of feedback in the system about prompting effectiveness, leaving users to grapple with the fuzzy abstraction matching problem without support”

Metacognition research suggests that increased effort requirements may discourage users from appropriate evaluation.

“How this will affect users' selfconfidence and AI reliance remains unclear, but metacognition research suggests that increased effort requirements may discourage users from appropriate evaluation [4].”

• cites [4] 4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1-15. sourceNot checked — no full text of the source

Although self-awareness and task decomposition are often required to some extent when interacting with GenAI systems, their pertinence increases as users concretize their usage intentions.

“We note that, although self-awareness and task decomposition are often required to some extent when interacting with GenAI systems, their pertinence increases as users concretize their usage intentions (e.g., achieving work or personal goals).”

Lu and Yin [108] show that, in the absence of AI accuracy information, humans rely on their agreement with AI as a heuristic for their reliance on it, albeit only when they have a high self-confidence.

“Lu and Yin [108] show that, in the absence of AI accuracy information, humans rely on their agreement with AI as a heuristic for their reliance on it, albeit only when they have a high self-confidence.”

• cites [108] 108. Zhuoran Lu and Ming Yin. 2021. Human Reliance on Machine Learning Models When Performance Feedback is Limited: Heuristics and Risks. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI '21). Association for Computing Machinery, New York, NY, USA, 1-16. https://doi. org/10.1145/3411764.3445562Not checked — no full text of the source

Evaluating and relying on AI output requires users to maintain a well-adjusted confidence in their own domain expertise and ability to evaluate output.

“Evaluating and relying on AI output requires users to maintain a well-adjusted confidence in their own domain expertise and ability to evaluate output (i.e., self-confidence; Figure 2c).”

GenAI exacerbates the demand for a well-adjusted confidence in output evaluation.

“We posit that GenAI exacerbates the demand for a well-adjusted confidence in output evaluation.”

• AssumptionRests on an assumption: Analogous research in GenAI that explicitly measures and manipulates user self-confidence is missing.

Recent research on AI-assisted decision-making finds that users' self-confidence is a key determinant of their reliance on AI responses.

“The importance of this metacognitive demand is evidenced in recent research on AIassisted decision-making, which finds that users' self-confidence is a key determinant of their reliance on AI responses”

Such interaction modes may arguably make output evaluation more challenging due to the need to infer the intent behind systems' suggestions.

“In fact, such interaction modes may arguably make output evaluation more challenging due to the need to infer the intent behind systems' suggestions [74].”

• cites [74] 74. Ken Gu, Madeleine Grunde-McLaughlin, Andrew M. McNutt, Jeffrey Heer, and Tim Althoff. 2023. How Do Data Analysts Respond to AI Assistance? A Wizard- of-Oz Study. https://doi.org/10.48550/arXiv.2309.10108 arXiv:2309.10108 [cs]. sourceNot checked — no full text of the source

Expert programmers have advanced metacognitive monitoring and control, in that they are able to identify their specific goals and decompose them into concrete tasks.

“Expert programmers have advanced metacognitive monitoring and control, in that they are able to identify their specific goals and decompose them into concrete tasks [60].”

• cites [60] 60. Anneli Eteläpelto. 1993. Metacognition and the Expertise of Computer Pro- gram Comprehension. Scandinavian Journal of Educational Research 37, 3 (Jan. 1993), 243-254. https://doi.org/10.1080/0031383930370305 Publisher: Routledge _eprint: https://doi.org/10.1080/0031383930370305. sourceNot checked — no full text of the source

The relative ease with which GenAI can produce extensive output may affect output evaluation and reliance via potentially misleading cues that people implicitly rely on to update their confidence and guide their subsequent metacognitive control.

“The relative ease with which GenAI can produce extensive output may also affect output evaluation and reliance via potentially misleading cues that people implicitly rely on to update their confidence and guide their subsequent metacognitive control [4].”

• cites [4] 4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1-15. sourceNot checked — no full text of the source

Novices in a domain or in GenAI may be particularly vulnerable to output evaluation challenges.

“Novices in a domain or in GenAI may be particularly vulnerable, as one participant commented on long code suggestions:”

Self-evaluation interventions can help users reflect on their task goals and approaches to task decomposition.

“self-evaluation interventions can help users reflect on their task goals and approaches to task decomposition [74].”

• cites [74] 74. Ken Gu, Madeleine Grunde-McLaughlin, Andrew M. McNutt, Jeffrey Heer, and Tim Althoff. 2023. How Do Data Analysts Respond to AI Assistance? A Wizard- of-Oz Study. https://doi.org/10.48550/arXiv.2309.10108 arXiv:2309.10108 [cs]. sourceNot checked — no full text of the source

Such an understanding can mitigate the risks of inappropriate evaluation of, and confidence in, AI-generated output.

“Such an understanding can in turn mitigate the risks of inappropriate evaluation of, and confidence in, AI-generated output.”

Self-evaluation can be used to support effective prompting.

“Self-evaluation can be used to support effective prompting.”

Changes in confidence can affect people's approach to evaluating GenAI output.

“More importantly, changes in confidence can affect people's approach to evaluating GenAI output.”

The benefits of generated content may also be diffuse and indirect.

“The benefits of generated content may also be diffuse and indirect.”

More flexible interfaces can support the crafting of a complex prompting strategy by enabling open-ended exploration of task goals and the relationships between them during the planning process.

“More flexible interfaces can support the crafting of a complex prompting strategy by enabling open-ended exploration of task goals and the relationships between them during the planning process.”

Tasks may be complex or ambiguous, often requiring gathering, organizing, and synthesizing information in a nonlinear manner distinct from the linear conversational interfaces in most GenAI systems today.

“However, tasks may nevertheless be complex or ambiguous, often requiring gathering, organizing, and synthesizing information in a nonlinear manner distinct from the linear conversational interfaces in most GenAI systems today.”

Participants co-writing with an LLM found that seeing the LLM's suggestions was helpful even when they did not implement them.

“For example, participants co-writing with an LLM found that seeing the LLM's suggestions was helpful even when they did not implement them [207].”

• cites [207] 207. Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces (IUI '22). Association for Computing Machinery, New York, NY, USA, 841-852. https://doi.org/10.1145/3490099.3511105 sourceNot checked — no full text of the source

Adjusting confidence in output evaluation typically requires objective measures of performance for comparison.

“Adjusting confidence in output evaluation typically requires objective measures of performance for comparison (e.g., the number of errors a user correctly detected in the output)”

By increasing people's confidence, such cues can affect their metacognitive control, leading people to decrease the effort they invest into further deliberate processing, as measured by thinking time and changes-of-mind.

“By increasing people's confidence, such cues can affect their metacognitive control, leading people to decrease the effort they invest into further deliberate processing, as measured by thinking time and changes-of-mind [4,183,184].”

• cites [4] 4. Rakefet Ackerman and Valerie Thompson. 2017. Meta-Reasoning: Shedding meta-cognitive light on reasoning research. 1-15. sourceNot checked — no full text of the source
• cites [183] 183. Valerie Thompson and Kinga Morsanyi. 2012. Analytic thinking: do you feel like it? Mind & Society 11, 1 (June 2012), 93-105. https://doi.org/10.1007/s11299- 012-0100-6 sourceNot checked — no full text of the source
• cites [184] 184. Valerie A. Thompson, Jamie A. Prowse Turner, Gordon Pennycook, Linden J. Ball, Hannah Brack, Yael Ophir, and Rakefet Ackerman. 2013. The role of answer fluency and perceptual fluency as metacognitive cues for initiating analytic thinking. Cognition 128, 2 (Aug. 2013), 237-251. https://doi.org/10.1016/j. cognition.2012.09.012 sourceNot checked — no full text of the source

Planning-related interventions can support users directly in task decomposition, improving prompt effectiveness via more explicit and discrete instructions.

“Planning-related interventions can also support users directly in task decomposition, improving prompt effectiveness via more explicit and discrete instructions.”

When applied to the prompting and output evaluation process itself, these approaches can make users aware of where they are in a task.

“When applied to the prompting and output evaluation process itself, these approaches can make users aware of where they are in a task.”

It is plausible that, although cognitive load may increase due to the processing requirements of many metacognitive support interventions, the improvement in metacognition may be accompanied by a simultaneous and larger reduction in cognitive load, resulting in a net decrease in cognitive load.

“The relationship between metacognition and cognitive load is an active research area [46,163,193,194], but it is plausible that, although cognitive load may increase due to the processing requirements of many metacognitive support interventions, the improvement in metacognition may be accompanied by a simultaneous and larger reduction in cognitive load, resulting in a net decrease in cognitive”

By providing contextual and performance information alongside the system inputs and outputs, explainability should help partly offload metacognitive processing from users' minds and onto the system interface.

“From the perspective of metacognitive demand, by providing contextual and performance information alongside the system inputs and outputs, explainability should help partly offload metacognitive processing from users' minds and onto the system interface.”

Increasing customizability can increase the demand for self-awareness, well-adjusted confidence, task decomposition, and metacognitive flexibility.

“On one hand, increasing customizability can increase the demand for self-awareness (e.g., “are any of the settings relevant to my task?”), well-adjusted confidence (e.g., “did any of the settings affect the output quality, or is it related to my prompting?”),task decomposition (e.g., “what is the right order to adjust settings for each of my subtasks?”),and metacognitive flexibility (e.g., “which”

Support getting started is a key user request for GenAI explainability.

“Support getting started is a key user request for GenAI explainability; see §4.2.1.”

• cites §4.2.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

All of the features that distinguish GenAI systems from search engines place unique demands on users' metacognition.

“As we aim to demonstrate here, all of these features place unique demands on users' metacognition and inform the design space of solutions to address these demands, a design space which necessarily extends beyond that of current search engines.”

• AssumptionRests on an assumption: GenAI systems are more flexible than search engines in their responsiveness to user prompts, in the range of implicit and explicit parameters available to users in their prompts, and in the multi-modality of their input/output space.
• AssumptionRests on an assumption: GenAI systems function as general-purpose tools, able to perform content generation, discrimination, and editing, unlike search engines.
• AssumptionRests on an assumption: Current GenAI systems are non-deterministic in their responses, unlike search engines.

Increasing customizability may increase cognitive load, particularly for novice users.

“This may increase cognitive load, particularly for novice users.”

Increased customizability can support metacognition, particularly for more advanced users.

“On the other hand, increased customizability can support metacognition, particularly for more advanced users.”

The design space of solutions to address these metacognitive demands necessarily extends beyond that of current search engines.

“As we aim to demonstrate here, all of these features place unique demands on users' metacognition and inform the design space of solutions to address these demands, a design space which necessarily extends beyond that of current search engines.”

Processing fluency can lead to inflated confidence without necessarily improving objective accuracy.

“As per §3.2, processing fluency can lead to inflated confidence without necessarily improving objective accuracy.”

• cites §3.2Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.

The usability challenges of prompting make it a key target for explainability.

“The usability challenges of prompting make it a key target for explainability, as per §4.2.1.”

• cites §4.2.1Not matched

  Couldn’t be matched to a paper in Crossref, so it can’t be checked.