"It's like a rubber duck that talks back": Understanding Generative AI-Assisted Data Analysis Workflows through a Participatory Prompting Study
Open accessThe DOI matches its Crossref record; 57 of the 58 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Generative AI tools can help users with many tasks. One such task is data analysis, which is notoriously challenging for non-expert end-users due to its expertise requirements, and where AI holds much potential, such as finding relevant data sources, proposing analysis strategies, and writing analysis code. To understand how data analysis workflows can be assisted or impaired by generative AI, we conducted a study (n=15) using Bing Chat via participatory prompting. Participatory prompting is a recently developed methodology in which users and researchers reflect together on tasks through co-engagement with generative AI. In this paper we demonstrate the value of the participatory prompting method. We found that generative AI benefits the information foraging and sensemaking loops of data analysis in specific ways, but also introduces its own barriers and challenges, arising from the difficulties of query formulation, specifying context, and verifying results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“We find that vibe coding follows iterative goal satisfaction cycles where developers alternate between prompting AI, evaluating generated code through rapid scanning and application testing, and manual editing.”
From Vibe coding: programming through conversation with artificial intelligence (2025) · cites this paper as Drosos et al., 2024
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“The direction of the project was significantly shaped by the back-and-forth with the system as the generated code variously: succeeded in matching the creator's intent directly, thus allowing progress; or succeeded indirectly such as by interpreting vague instructions, thus sparking ideas; or failed, thus necessitating debugging or a goal 'pivot' (Drosos et al., 2024).”
From Vibe coding: programming through conversation with artificial intelligence (2025) · cites this paper as Drosos et al., 2024
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“In line with recent projections that more accessible GenAI tools may exacerbate the risks of technology over-reliance [29,102,130]”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [29]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Prior work has identified the work of understanding and evaluating GenAI output as a key aspect of working with GenAI [29,46,114,130].”
From The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers (2025) · cites this paper as [29]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Gioia, D. A., Mehra, A., & Weick, K. E. (1996) · The Academy of Management Review · doi:10.2307/259169 · Cited by 15,614
(1969) · Virtual Defense Library (Ministerio de Defensa) · Cited by 10,735
Survey of Hallucination in Natural Language Generation
Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y. J., Madotto, A., & Fung, P. (2023) · ACM Computing Surveys · doi:10.1145/3571730 · Cited by 5,025
Contextual Design: Defining Customer-Centered Systems
Beyer, H. R., & Holtzblatt, K. (1997) · Cited by 3,229
Direct Manipulation: A Step Beyond Programming Languages
Shneiderman (1983) · Computer · doi:10.1109/mc.1983.1654471 · Cited by 1,939 · cited 1×
“We believe that the method can be extended to other domains, and here make five suggestions for fundamental aspects of the method that researchers should consider.”
MethodWhy Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
McDonald, N., Schoenebeck, S., & Forte, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359174 · Cited by 1,206
Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021) · arXiv · doi:10.48550/arxiv.2102.09692 · Cited by 1,124
Out of One, Many: Using Language Models to Simulate Human Samples
Argyle, L. P., Busby, E. C., Fulda, N., Gubler, J. R., Rytting, C., & Wingate, D. (2023) · Political Analysis · doi:10.1017/pan.2023.2 · Cited by 783
The cost structure of sensemaking
Russell, D. M., Stefik, M. J., Pirolli, P., & Card, S. K. (1993) · Proceedings of the SIGCHI conference on Human factors in computing systems - CHI '93 · doi:10.1145/169059.169209 · Cited by 726
The state of the art in end-user software engineering
Ko, A. J., Abraham, R., Beckwith, L., Blackwell, A., Burnett, M., Erwig, M., Scaffidi, C., Lawrance, J., Lieberman, H., Myers, B., et al. (2011) · ACM Computing Surveys · doi:10.1145/1922649.1922658 · Cited by 633
Automating string processing in spreadsheets using input-output examples
Gulwani, S. (2011) · ACM SIGPLAN Notices · doi:10.1145/1925844.1926423 · Cited by 581
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436
Explanations Can Reduce Overreliance on AI Systems During Decision-Making
Vasconcelos, H., Jörke, M., Grunde-McLaughlin, M., Gerstenberg, T., Bernstein, M. S., & Krishna, R. (2023) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3579605 · Cited by 402
Automatic Prompt Optimization with “Gradient Descent” and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C., & Zeng, M. (2023) · Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2023.emnlp-main.494 · Cited by 191
Automatic Prompt Optimization with "Gradient Descent" and Beam Search
Pryzant, R., Iter, D., Li, J., Lee, Y. T., Zhu, C., & Zeng, M. (2023) · arXiv · doi:10.48550/arxiv.2305.03495 · Cited by 191
Handbook of Human Computer Interaction
(2021) · Springer eBooks · doi:10.1007/978-3-319-27648-9 · Cited by 166
A user-centred approach to functions in Excel
Jones, S., Blackwell, A. F., & Burnett, M. (2003) · ACM SIGPLAN International Conference on Functional Programming (ICFP) · doi:10.1145/944705.944721 · Cited by 165
Lee, S., Kim, S.-H., Hung, Y.-H., Lam, H., Kang, Y.-A., & Yi, J. S. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2015.2467195 · Cited by 161
Semnani, S. J., Yao, V., Zhang, H., & Lam, M. S. (2023) · Findings of the Association for Computational Linguistics: EMNLP (Findings of EMNLP) · doi:10.18653/v1/2023.findings-emnlp.157 · Cited by 146
Gmeiner, F., Yang, H., Yao, L., Holstein, K., & Martelaro, N. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580999 · Cited by 130
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
ClassSheets: automatic generation of spreadsheet applications from object-oriented specifications
Engels, G., & Erwig, M. (2005) · IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1145/1101908.1101929 · Cited by 128
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580817 · Cited by 119
Self-citationA survey of intelligent assistants for data analysis
Serban, F., Vanschoren, J., Kietz, J.-U., & Bernstein, A. (2013) · ACM Computing Surveys · doi:10.1145/2480741.2480748 · Cited by 111
Dang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83
On the Design of AI-powered Code Assistants for Notebooks
Mcnutt, A. M., Wang, C., Deline, R. A., & Drucker, S. M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 77
AutoDS: Towards Human-Centered Automation of Data Science
Wang, D., Andres, J., Weisz, J. D., Oduor, E., & Dugan, C. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445526 · Cited by 75
Spreadsheet Comprehension: Guesswork, Giving Up and Going Back to the Author
Srinivasa Ragavan, S., Sarkar, A., & Gordon, A. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445634 · Cited by 67
Self-citationEnd-user debugging strategies: A sensemaking perspective
Grigoreanu, V., Burnett, M., Wiedenbeck, S., Cao, J., Rector, K., & Kwan, I. (2012) · ACM Transactions on Computer-Human Interaction · doi:10.1145/2147783.2147788 · Cited by 60
Sarkar, A. (2023) · Proceedings of the 2nd Annual Meeting of the Symposium on Human-Computer Interaction for Work · doi:10.1145/3596671.3597650 · Cited by 50
Self-citationEnough With “Human-AI Collaboration”
Sarkar, A. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3582735 · Cited by 47
Self-citationWhat you see is what you test: a methodology for testing form-based visual programs
Rothermel, G., Li, L., DuPuis, C., & Burnett, M. (2002) · Proceedings of the 20th International Conference on Software Engineering · doi:10.1109/icse.1998.671118 · Cited by 43
Exploring the Learnability of Program Synthesizers by Novice Programmers
Jayagopal, D., Lubin, J., & Chasins, S. E. (2022) · Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3526113.3545659 · Cited by 40
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641891 · Cited by 40
Chalhoub, G., & Sarkar, A. (2022) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3491102.3501833 · Cited by 38
Self-citationHow Do Analysts Understand and Verify AI-Assisted Data Analyses?
Gu, K., Shang, R., Althoff, T., Wang, C., & Drucker, S. M. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642497 · Cited by 37
Li, X., Zhang, Y., Leung, J., Sun, C., & Zhao, J. (2023) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3545995 · Cited by 36
Using Annotations for Sensemaking About Code
Horvath, A., Myers, B. A., Macvean, A., & Rahman, I. (2022) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3526113.3545667 · Cited by 32
TweakIt: Supporting End-User Programmers Who Transmogrify Code
Lau, S., Ragavan, S. S., Milne, K., Barik, T., & Sarkar, A. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445265 · Cited by 30
Self-citationConstructivist Design for Interactive Machine Learning
Sarkar, A. (2016) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/2851581.2892547 · Cited by 25
Self-citationECOVAL: Ecological Validity of Cues and Representative Design in User Experience Evaluations
de Louvain, U. C., & Kieffer, S. (2017) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00093 · Cited by 25
Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · Proceedings of the 2023 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software · doi:10.1145/3622758.3622882 · Cited by 23
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23
Should Computers Be Easy To Use? Questioning the Doctrine of Simplicity in User Interface Design
Sarkar, A. (2023) · Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544549.3582741 · Cited by 19
Self-citationBeyond Visual Analytics: Human-Machine Teaming for AI-Driven Data Sensemaking
Wenskovitch, J. E., Fallon, C. K., Miller, K. H., & Dasgupta, A. (2021) · IEEE Workshop on TRust and EXpertise in Visual Analytics (TREX) · doi:10.1109/trex53765.2021.00012 · Cited by 19
End-user encounters with lambda abstraction in spreadsheets: Apollo’s bow or Achilles’ heel?
Sarkar, A., Ragavan, S. S., Williams, J., & Gordon, A. D. (2022) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc53370.2022.9833131 · Cited by 18
Self-citationInteractive visual machine learning in spreadsheets
Sarkar, A., Jamnik, M., Blackwell, A. F., & Spott, M. (2015) · 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2015.7357211 · Cited by 17
Self-citationCOLDECO: An End User Spreadsheet Inspection Tool for AI-Generated Code
Ferdowsi, K., Williams, J., Drosos, I., Gordon, A. D., Negreanu, C., Polikarpova, N., Sarkar, A., & Zorn, B. (2023) · 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl-hcc57772.2023.00017 · Cited by 17
Self-citationElastic sheet-defined functions: Generalising spreadsheet functions to variable-size input arrays
McCutchen, M., Borghouts, J. W., Gordon, A. D., Jones, S., & Sarkar, A. (2020) · Journal of Functional Programming · doi:10.1017/s0956796820000234 · Cited by 16
Self-citationDo We Need Natural Language? Exploring Restricted Language Interfaces for Complex Domains
Mu, J., & Sarkar, A. (2019) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3290607.3312975 · Cited by 16
Self-citationTeach and try: A simple interaction technique for exploratory data modelling by end users
Sarkar, A., Blackwell, A. F., Jamnik, M., & Spott, M. (2014) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2014.6883022 · Cited by 14
Self-citationGridlets: Reusing Spreadsheet Grids
Joharizadeh, N., Sarkar, A., Gordon, A. D., & Williams, J. M. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382806 · Cited by 14
Self-citationUnderstanding and Inferring Units in Spreadsheets
Williams, J. M., Negreanu, C., Gordon, A. D., & Sarkar, A. (2020) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc50065.2020.9127254 · Cited by 14
Self-citationVisual discovery and model-driven explanation of time series patterns
Sarkar, A., Spott, M., Blackwell, A. F., & Jamnik, M. (2016) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vlhcc.2016.7739668 · Cited by 13
Self-citationSpreadsheet Use and Programming Experience: An Exploratory Survey
Sarkar, A., Borghouts, J. W., Iyer, A., Khullar, S., Canton, C., Hermans, F., Gordon, A. D., & Williams, J. M. (2020) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3334480.3382807 · Cited by 13
Self-citationCollaborative Human-AI Sensemaking for Intelligence Analysis
Dorton, S. L., & Hall, R. A. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-77772-2_12 · Cited by 13
Consciousness and Emotion in Cognitive Science: Conceptual and Empirical Issues
Toribio, J., & Clark, A. (1998) · Routledge eBooks · doi:10.4324/9780203826430 · Cited by 6
Research works printed but not linked: 29
- Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
- Tingfeng Cao, Chengyu Wang, Bingyan Liu, Ziheng Wu, Jinhui Zhu, and Jun Huang. 2023. BeautifulPrompt: Towards Automatic Prompt Engineering for Text-to-Image Synthesis. In Conference on Empirical Methods in Natural Language Processing. https://api.semanticscholar.org/CorpusID:265150243
- Souti Chattopadhyay, Zixuan Feng, Emily Arteaga, Audrey Au, Gonzalo Ramos, Titus Barik, and Anita Sarma. 2023. Make It Make Sense! Understanding and Facilitating Sensemaking in Computational Notebooks. arXiv preprint arXiv:2312.11431 (2023). · arXiv
- Chen Chen, Jane Hoffswell, Shunan Guo, Ryan Rossi, Yeuk-Yin Chan, Fan Du, Eunyee Koh, and Zhicheng Liu. 2023. WHATSNEXT: Guidance-enriched Exploratory Data Analysis with Interactive, Low-Code Notebooks. In 2023 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 209–214.
- Ewart J De Visser, Samuel S Monfort, Ryan McKendrick, Melissa AB Smith, Patrick E McKnight, Frank Krueger, and Raja Parasuraman. 2016. Almost human: Anthropomorphism increases trust resilience in cognitive agents.Journal of Experimental Psychology: Applied 22, 3 (2016), 331.
- Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023. Retrieval-Augmented Generation for Large Language Models: A Survey. ArXiv abs/2312.10997 (2023). https://api.semanticscholar.org/CorpusID:266359151 · arXiv
- Andrew D Gordon, Carina Negreanu, José Cambronero, Rasika Chakravarthy, Ian Drosos, Hao Fang, Bhaskar Mitra, Hannah Richardson, Advait Sarkar, Stephanie Simmons, 2023. Co-audit: tools to help humans double-check AI-generated content. arXiv preprint arXiv:2310.01297 (2023). · arXiv
- Theodore Jensen and Mohammad Maifi Hasan Khan. 2022. I’m Only Human: The Effects of Trust Dampening by Anthropomorphic Agents. In International Conference on Human-Computer Interaction. Springer, 285–306.
- Philip N Johnson-Laird and Keith Oatley. 1998. Basic emotions, rationality, and folk theory. In Consciousness and Emotion in Cognitive Science. Routledge, 289–311.
- John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
- Deborah Lupton. 2016. The quantified self. John Wiley & Sons.
- Mariana Mărăşoiu, Alan F Blackwell, Advait Sarkar, and Martin Spott. 2016. Clarifying hypotheses by sketching data. In Proceedings of the Eurographics/IEEE VGTC Conference on Visualization: Short Papers. 125–129.
- Samir Passi and Mihaela Vorvoreanu. 2022. Overreliance on AI: literature review. Microsoft Research (2022).
- Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643.
- Peter Pirolli and Stuart Card. 2005. The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. In Proceedings of international conference on intelligence analysis, Vol. 5. McLean, VA, USA, 2–4.
- James Prather, Brent N Reeves, Paul Denny, Brett A Becker, Juho Leinonen, Andrew Luxton-Reilly, Garrett Powell, James Finnie-Ansley, and Eddie Antonio Santos. 2023. " It’s Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers. arXiv preprint arXiv:2304.02491 (2023). · arXiv
- Advait Sarkar. 2016. Interactive analytical modelling. Technical Report UCAM-CL-TR-920. University of Cambridge Computer Laboratory. https://doi.org/10.48456/tr-920 · DOI
- Advait Sarkar. 2022. Is explainable AI a race against model complexity?. In Workshop on Transparency and Explanations in Smart Systems (TeXSS), in conjunction with ACM Intelligent User Interfaces (IUI 2022)(CEUR Workshop Proceedings, 3124). 192–199. http://ceur-ws.org/Vol-3124/paper22.pdf
- Advait Sarkar. 2024. AI Should Challenge, Not Obey. Communications of the ACM (in press) (2024).
- Advait Sarkar. 2024. Large Language Models Cannot Explain Themselves. In ACM CHI 2024 Workshop on Human-Centered Explainable AI (HCXAI).
- Advait Sarkar, Ian Drosos, Rob Deline, Andrew D. Gordon, Carina Negreanu, Sean Rintel, Jack Williams, and Ben Zorn. 2023. Participatory prompting: a user-centric research method for eliciting AI assistance opportunities in knowledge workflows. In Proceedings of the 34th Annual Conference of the Psychology of Programming Interest Group (PPIG 2023).
- Advait Sarkar and Andrew D. Gordon. 2018. How do people learn to use spreadsheets? (Work in progress). In Proceedings of the 29th Annual Conference of the Psychology of Programming Interest Group (PPIG 2018). 28–35.
- Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence?. In Proceedings of the 33rd Annual Conference of the Psychology of Programming Interest Group (PPIG 2022).
- Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press.
- Divya Siddarth, Daron Acemoglu, Danielle Allen, Kate Crawford, James Evans, Michael Jordan, and E Weyl. 2021. How AI fails us. arXiv preprint arXiv:2201.04200 (2021). · arXiv
- Robert Stalnaker. 2002. Common ground. Linguistics and philosophy 25, 5/6 (2002), 701–721.
- Lev Tankelevitch, Viktor Kewenig, Auste Simkute, Ava Elizabeth Scott, Advait Sarkar, Abigail Sellen, and Sean Rintel. 2023. The Metacognitive Demands and Opportunities of Generative AI. arXiv preprint arXiv:2312.10893 (2023). · arXiv
- Weixuan Wang, Barry Haddow, Alexandra Birch, and Wei Peng. 2023. Assessing the Reliability of Large Language Model Knowledge. ArXiv abs/2310.09820 (2023). https://api.semanticscholar.org/CorpusID:264146357 · arXiv
- Zilong Wang, Hao Zhang, Chun-Liang Li, Julian Martin Eisenschlos, Vincent Perot, Zifeng Wang, Lesly Miculicich, Yasuhisa Fujii, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2024. Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding. ArXiv abs/2401.04398 (2024). https://api.semanticscholar.org/CorpusID:266899992 · arXiv
Cited by
Lee, H.-P. (., Sarkar, A., Tankelevitch, L., Drosos, I., Rintel, S., Banks, R., & Wilson, N. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713778 · Cited by 490
Self-citationImproving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
Kazemitabaar, M., Williams, J., Drosos, I., Grossman, T., Henley, A. Z., Negreanu, C., & Sarkar, A. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676345 · Cited by 112
Self-citation"It makes you think": Provocations Help Restore Critical Thinking to AI-Assisted Knowledge WorkPreprint
Drosos, I., Sarkar, A., Xiaotong, Xu, & Toronto, N. (2025) · arXiv (Cornell University) · doi:10.48550/arxiv.2501.17247 · Cited by 43
Self-citationSarkar, A. (2024) · Communications of the ACM · doi:10.1145/3649404 · Cited by 40
Self-citationTankelevitch, L., Glassman, E. L., He, J., Kazemitabaar, M., Kittur, A., Lee, M., Palani, S., Sarkar, A., Ramos, G., Rogers, Y., et al. (2025) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3706599.3706745 · Cited by 37
Self-citation
Metadata
- Authors
- Drosos, I., Sarkar, A., Xu, X., Negreanu, C., Rintel, S., & Tankelevitch, L.
- Affiliations
- University of Cambridge (GB)
- Microsoft Research (United Kingdom) (GB)
- Venue
- Proceedings of the 3rd Annual Meeting of the Symposium on Human-Computer Interaction for Work
- Publisher
- ACM
- Published
- 2024-06-25
- Pages
- 1-21
- OpenAlex
- W4399913584
- Cited by
- 30
- References
- 54
- Citations by year
- 2026: 12 · 2025: 16 · 2024: 2
- FWCI
- 6.19
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- Annual Symposium on Human-Computer Interaction for Work (CHIWORK) (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- generative AI, data analysis pipeline, Bing Chat, information foraging, sensemaking, query formulation, context specification, result verification, AI-assisted programming
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.