The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.
How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study
Open accessAbstract
Data analysis is challenging as analysts must navigate nuanced decisions that may yield divergent conclusions. AI assistants have the potential to support analysts in planning their analyses, enabling more robust decision making. Though AI-based assistants that target code execution (e.g., Github Copilot) have received significant attention, limited research addresses assistance for both analysis execution and planning. In this work, we characterize helpful planning suggestions and their impacts on analysts’ workflows. We first review the analysis planning literature and crowd-sourced analysis studies to categorize suggestion content. We then conduct a Wizard-of-Oz study (n=13) to observe analysts’ preferences and reactions to planning assistance in a realistic scenario. Our findings highlight subtleties in contextual factors that impact suggestion helpfulness, emphasizing design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“The granularity of control might be high-level (strategic direction [61]) or fine-grained (parameter tuning [208]) depending on the phase.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [61]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Estimating the reproducibility of psychological science
Open Science Collaboration (2015) · Science · doi:10.1126/science.aac4716 · Cited by 8,515 · cited 2×
“Further, the unconsidered flexibility in decision-making (and the biases inherent therein) contributes to the scientific reproducibility crisis [1, 10, 33].”
BackgroundWohlin, C. (2014) · Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering · doi:10.1145/2601248.2601268 · Cited by 4,234 · cited 1×
“We searched Google Scholar with the keywords "sensemaking data analysis" from which we iteratively snowball sampled [117].”
Method1,500 scientists lift the lid on reproducibility
Baker, M. (2016) · Nature · doi:10.1038/533452a · Cited by 4,209 · cited 2×
“As data analysis often informs high-stakes real-world decisions [1, 9, 31], it is essential that analysts be able to understand and think critically about their analyses, regardless whether they received execution or planning assistance.”
BackgroundTraining language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
PaLM: Scaling Language Modeling with Pathways
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.02311 · Cited by 2,590
Guidelines for Human-AI Interaction
Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., et al. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300233 · Cited by 2,439
Box, G. E. P. (1976) · Journal of the American Statistical Association · doi:10.1080/01621459.1976.10480949 · Cited by 2,114
Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems
(2020) · doi:10.1145/3313831 · Cited by 1,951
Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy
Shneiderman, B. (2020) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1741118 · Cited by 1,743
Research works printed but not linked: 25
- Jacob Andreas. 2022. Language Models as Agent Models. ArXiv abs/2212.01681 (2022). · arXiv
- Leilani Battle and Jeffrey Heer. 2019. Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau. Computer Graphics Forum 38 (2019).
- Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv abs/2303.12712 (2023). · arXiv
- Katharina Buchholz. 2023. Time to One Million Users. https://www.statista.com/chart/29174/time-to-one-million-users/. Accessed June 5, 2023.
- Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and Evaluating a Jupyter Notebook Data Science Assistant. ArXiv abs/2201.12901 (2022). · arXiv
- Liying Cheng, Xingxuan Li, and Lidong Bing. 2023. Is GPT-4 a Good Data Analyst?ArXiv abs/2305.15038 (2023). · arXiv
- Victor C. Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. ArXiv abs/2303.02927 (2023). · arXiv
- Google. 2022. ML-Enhanced Code Completion Improves Developer Productivity. Google AI Blog. https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html Accessed: June 3, 2023.
- Andrew D Gordon, Carina Negreanu, José Cambronero, Rasika Chakravarthy, Ian Drosos, 2023. Co-audit: tools to help humans double-check AI-generated content. arXiv preprint arXiv:2310.01297 (2023). · arXiv
- Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven Mark Drucker. 2023. How Do Analysts Understand and Verify AI-Assisted Data Analyses?ArXiv abs/2309.10947 (2023). https://api.semanticscholar.org/CorpusID:262064320 · arXiv
- Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. ArXiv abs/2303.05453 (2023). · arXiv
- Jenny Liang, Chenyang Yang, and Brad A. Myers. 2023. Understanding the Usability of AI Programming Assistants. ArXiv abs/2303.17125 (2023). · arXiv
- Qingzi Vera Liao and S. Shyam Sundar. 2022. Designing for Responsible Trust in AI Systems: A Communication Perspective. 2022 ACM Conference on Fairness, Accountability, and Transparency (2022).
- Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Trans. Mach. Learn. Res. 2022 (2022). https://api.semanticscholar.org/CorpusID:249191391
- Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johanneson, 2021. Non-Standard Errors. SSRN Electronic Journal (2021).
- Swaroop Mishra and Elnaz Nouri. 2022. HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models. ArXiv abs/2208.08232 (2022). · arXiv
- Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, 2022. A Conversational Paradigm for Program Synthesis. ArXiv abs/2203.13474 (2022). · arXiv
- Fernando Pérez and Brian E. Granger. 2007. IPython: A System for Interactive Scientific Computing. Computing in Science & Engineering 9 (2007).
- Deepthi Raghunandan, Aayushi Roy, Shenzhi Shi, Niklas Elmqvist, and Leilani Battle. 2022. Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2022).
- Abhraneel Sarma, Alex Kale, Michael Moon, Nathan Taback, Fanny Chevalier, 2021. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2021). https://api.semanticscholar.org/CorpusID:238541538
- Uri Simonsohn, Joseph P. Simmons, and Leif D. Nelson. 2015. Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications. MKTG: Methodological Issues in Consumer Research (Topic) (2015).
- Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones 2017. Attention is All you Need. In NIPS.
- Chris J. Wild and Maxine Pfannkuch. 1999. Statistical Thinking in Empirical Enquiry. International Statistical Review 67 (1999).
- Kanit Wongsuphasawat, Yang Liu, and Jeffrey Heer. 2019. Goals, Process, and Challenges of Exploratory Data Analysis: An Interview Study. ArXiv abs/1911.00568 (2019). · arXiv
- Beiqi Zhang, Peng Liang, Xiyu (Thomas) Zhou, Aakash Ahmad, and Muhammad Waseem. 2023. Practices and Challenges of Using GitHub Copilot: An Empirical Study. ArXiv abs/2303.08733 (2023). · arXiv
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
“Correlation between confidence and objective accuracy can estimate confidence calibration [127, 202]; Meta-d’ is a derived metric capturing users’ prospective confidence resolution independent of their calibration [66, 115] (see also [65, 145]).”
BackgroundMethodData-Copilot: Bridging Billions of Data and Humans with Autonomous WorkflowPreprint
Zhang, W., Shen, Y., Tan, Z., Hou, G., Lu, W., & Zhuang, Y. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2306.07209 · Cited by 122
Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
Kazemitabaar, M., Williams, J., Drosos, I., Grossman, T., Henley, A. Z., Negreanu, C., & Sarkar, A. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676345 · Cited by 111
Kazemitabaar, M., Huang, O., Suh, S., Henley, A. Z., & Grossman, T. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712104 · Cited by 92
Ironies of Generative AI: Understanding and Mitigating Productivity Loss in Human-AI Interaction
Simkute, A., Tankelevitch, L., Kewenig, V., Scott, A. E., Sellen, A., & Rintel, S. (2024) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2024.2405782 · Cited by 89
“Otherwise, during this stage, suggestions should carefully align with users’ flow (Gu et al., 2023), in line with ecological interface design.”
Influential
Metadata
- Authors
- Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T.
- Affiliations
- University of Washington (US)
- Venue
- Proceedings of the CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2024-05-11
- Pages
- 1-22
- OpenAlex
- W4396832182
- Semantic Scholar
- 2fe74a07051649cb5dc41da319fef547647e9171
- Cited by
- 40
- References
- 94
- Citations by year
- 2026: 17 · 2025: 19 · 2024: 4
- FWCI
- 8.26
- Type
- conference paper
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- OSF Preprints (OSF Preprints) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Funders
- Bill & Melinda Gates Foundation (INV-004841)
- NSF (IIS-1901386)
- NSF CAREER (IIS-2142794)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
- Keywords
- AI assistants, data analysis, AI-assisted decision-making
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. https://app.cochat.ai/papers/10.1145/3613904.3641891
- The paper
- Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024). How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641891
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.