Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 99 of the 99 references it cites were found in a registry or an index.

How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study

Open access
Gu, Ken, Grunde-McLaughlin, Madeleine, McNutt, Andrew, Heer, Jeffrey, Althoff, Tim · 2024 · Proceedings of the CHI Conference on Human Factors in Computing Systems

Abstract

Data analysis is challenging as analysts must navigate nuanced decisions that may yield divergent conclusions. AI assistants have the potential to support analysts in planning their analyses, enabling more robust decision making. Though AI-based assistants that target code execution (e.g., Github Copilot) have received significant attention, limited research addresses assistance for both analysis execution and planning. In this work, we characterize helpful planning suggestions and their impacts on analysts’ workflows. We first review the analysis planning literature and crowd-sourced analysis studies to categorize suggestion content. We then conduct a Wizard-of-Oz study (n=13) to observe analysts’ preferences and reactions to planning assistance in a realistic scenario. Our findings highlight subtleties in contextual factors that impact suggestion helpfulness, emphasizing design implications for supporting different abstractions of assistance, forms of initiative, increased engagement, and alignment of goals between analysts and assistants.

Overview

Integrity signals

Registry

Crossref · 10.1145/3613904.3641891

Cites

99 of 99 references verified0 citation errors · 0 not checkable

Statistics

Recomputing p-values needs the paper’s full text.

Verification

Claims the paper makes

We couldn’t retrieve this paper’s full text, so its claims and self-check weren’t run. The registry and citation checks still apply.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Citations

Cites

Show all 94 references

Research works printed but not linked: 25

  1. Jacob Andreas. 2022. Language Models as Agent Models. ArXiv abs/2212.01681 (2022). · arXiv
  2. Leilani Battle and Jeffrey Heer. 2019. Characterizing Exploratory Visual Analysis: A Literature Review and Evaluation of Analytic Provenance in Tableau. Computer Graphics Forum 38 (2019).
  3. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, 2023. Sparks of Artificial General Intelligence: Early experiments with GPT-4. ArXiv abs/2303.12712 (2023). · arXiv
  4. Katharina Buchholz. 2023. Time to One Million Users. https://www.statista.com/chart/29174/time-to-one-million-users/. Accessed June 5, 2023.
  5. Shubham Chandel, Colin B. Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and Evaluating a Jupyter Notebook Data Science Assistant. ArXiv abs/2201.12901 (2022). · arXiv
  6. Liying Cheng, Xingxuan Li, and Lidong Bing. 2023. Is GPT-4 a Good Data Analyst?ArXiv abs/2305.15038 (2023). · arXiv
  7. Victor C. Dibia. 2023. LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models. ArXiv abs/2303.02927 (2023). · arXiv
  8. Google. 2022. ML-Enhanced Code Completion Improves Developer Productivity. Google AI Blog. https://ai.googleblog.com/2022/07/ml-enhanced-code-completion-improves.html Accessed: June 3, 2023.
  9. Andrew D Gordon, Carina Negreanu, José Cambronero, Rasika Chakravarthy, Ian Drosos, 2023. Co-audit: tools to help humans double-check AI-generated content. arXiv preprint arXiv:2310.01297 (2023). · arXiv
  10. Ken Gu, Ruoxi Shang, Tim Althoff, Chenglong Wang, and Steven Mark Drucker. 2023. How Do Analysts Understand and Verify AI-Assisted Data Analyses?ArXiv abs/2309.10947 (2023). https://api.semanticscholar.org/CorpusID:262064320 · arXiv
  11. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. ArXiv abs/2303.05453 (2023). · arXiv
  12. Jenny Liang, Chenyang Yang, and Brad A. Myers. 2023. Understanding the Usability of AI Programming Assistants. ArXiv abs/2303.17125 (2023). · arXiv
  13. Qingzi Vera Liao and S. Shyam Sundar. 2022. Designing for Responsible Trust in AI Systems: A Communication Perspective. 2022 ACM Conference on Fairness, Accountability, and Transparency (2022).
  14. Stephanie C. Lin, Jacob Hilton, and Owain Evans. 2022. Teaching Models to Express Their Uncertainty in Words. Trans. Mach. Learn. Res. 2022 (2022). https://api.semanticscholar.org/CorpusID:249191391
  15. Albert J. Menkveld, Anna Dreber, Felix Holzmeister, Juergen Huber, Magnus Johanneson, 2021. Non-Standard Errors. SSRN Electronic Journal (2021).
  16. Swaroop Mishra and Elnaz Nouri. 2022. HELP ME THINK: A Simple Prompting Strategy for Non-experts to Create Customized Content with Models. ArXiv abs/2208.08232 (2022). · arXiv
  17. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, 2022. A Conversational Paradigm for Program Synthesis. ArXiv abs/2203.13474 (2022). · arXiv
  18. Fernando Pérez and Brian E. Granger. 2007. IPython: A System for Interactive Scientific Computing. Computing in Science & Engineering 9 (2007).
  19. Deepthi Raghunandan, Aayushi Roy, Shenzhi Shi, Niklas Elmqvist, and Leilani Battle. 2022. Code Code Evolution: Understanding How People Change Data Science Notebooks Over Time. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2022).
  20. Abhraneel Sarma, Alex Kale, Michael Moon, Nathan Taback, Fanny Chevalier, 2021. multiverse: Multiplexing Alternative Data Analyses in R Notebooks. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (2021). https://api.semanticscholar.org/CorpusID:238541538
  21. Uri Simonsohn, Joseph P. Simmons, and Leif D. Nelson. 2015. Specification Curve: Descriptive and Inferential Statistics on All Reasonable Specifications. MKTG: Methodological Issues in Consumer Research (Topic) (2015).
  22. Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones 2017. Attention is All you Need. In NIPS.
  23. Chris J. Wild and Maxine Pfannkuch. 1999. Statistical Thinking in Empirical Enquiry. International Statistical Review 67 (1999).
  24. Kanit Wongsuphasawat, Yang Liu, and Jeffrey Heer. 2019. Goals, Process, and Challenges of Exploratory Data Analysis: An Interview Study. ArXiv abs/1911.00568 (2019). · arXiv
  25. Beiqi Zhang, Peng Liang, Xiyu (Thomas) Zhou, Aakash Ahmad, and Muhammad Waseem. 2023. Practices and Challenges of Using GitHub Copilot: An Empirical Study. ArXiv abs/2303.08733 (2023). · arXiv

Cited by

See all 75

Metadata

Authors
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T.
Affiliations
University of Washington (US)
Venue
Proceedings of the CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2024-05-11
Pages
1-22
OpenAlex
W4396832182
Cited by
40
References
94
Citations by year
2026: 17 · 2025: 19 · 2024: 4
FWCI
8.26
Type
conference paper
Language
en
Open access
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion, cc-by)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • OSF Preprints (OSF Preprints) (submittedVersion)
  • arXiv (Cornell University) (cc-by)
Funders
  • Bill & Melinda Gates Foundation (INV-004841)
  • NSF (IIS-1901386)
  • NSF CAREER (IIS-2142794)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Information Systems › Recommender Systems and Techniques
Keywords
AI assistants, data analysis, AI-assisted decision-making

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. https://app.cochat.ai/papers/10.1145/3613904.3641891
The paper
Gu, K., Grunde-McLaughlin, M., McNutt, A., Heer, J., & Althoff, T. (2024). How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study. Proceedings of the CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3613904.3641891

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.