CoChat

WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization

Open access
Xie, Liwenhan, Zheng, Chengbo, Xia, Haijun, Qu, Huamin, Zhu-Tian, Chen · 2024 · Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology

The DOI matches its Crossref record; the references it cites are not checked yet.

Abstract

Large language models (LLMs) support data analysis through conversational user interfaces, as exemplified in OpenAI’s ChatGPT (formally known as Advanced Data Analysis or Code Interpreter). Essentially, LLMs produce code for accomplishing diverse analysis tasks. However, presenting raw code can obscure the logic and hinder user verification. To empower users with enhanced comprehension and augmented control over analysis conducted by LLMs, we propose a novel approach to transform LLM-generated code into an interactive visual representation. In the approach, users are provided with a clear, step-by-step visualization of the LLM-generated code in real time, allowing them to understand, verify, and modify individual data operations in the analysis. Our design decisions are informed by a formative study (N=8) probing into user practice and challenges. We further developed a prototype named WaitGPT and conducted a user study (N=12) to evaluate its usability and effectiveness. The findings from the user study reveal that WaitGPT facilitates monitoring and steering of data analysis performed by LLMs, enabling participants to enhance error detection and increase their overall confidence in the results.

Overview

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3654777.3676374

Cites

Statistics

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

The citation lists are queued

They are fetched later and appear here then.

References 61 · Cited by 52

Research works printed but not linked: 10

  1. Yida Chen Aoyu Wu Catherine Yeh Trevor DePodesta Kenneth Li Nicholas Castillo Marin Oam Patel Jan Riecke Shivam Raval Olivia Seow Martin Wattenberg and Fernanda Viégas. 2024. Designing a dashboard for transparency and control of conversational AI. arXiv:2406.07882 · arXiv
  2. Google. 2024. Gemini Advanced: Release updates. Google. Retrieved June 1, 2024 from https://gemini.google.com/updates2024.05.21: updates on data analysis features.
  3. Yi Guo, Nan Cao, Xiaoyu Qi, Haoyang Li, Danqing Shi, Jing Zhang, Qing Chen, and Daniel Weiskopf. 2023. Urania: Visualizing data analysis pipelines for natural language-based data exploration. arXiv:2306.07760 · arXiv
  4. Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. arXiv:2407.02651 · arXiv
  5. Boyan Li Yuyu Luo Chengliang Chai Guoliang Li and Nan Tang. 2024. The dawn of natural language to SQL: Are we fully ready?arXiv:2406.01265 · arXiv
  6. Hua Shen Tiffany Knearem Reshmi Ghosh Kenan Alkiek Kundan Krishna Yachuan Liu Ziqiao Ma Savvas Petridis Yi-Hao Peng Li Qiwei 2024. Towards bidirectional human-AI alignment: A systematic review for clarifications framework and future directions. arXiv:2406.09264 · arXiv
  7. Bret Victor. 2011. Up and down the ladder of abstraction: A systematic approach to interactive visualization. Retrieved April 1, 2024 from http://worrydream.com/LadderOfAbstraction/
  8. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better LLM agents. In Proceedings of the International Conference on Machine Learning (ICML). Article PMLR 235, 13 pages. arXiv:2402.01030 · arXiv
  9. Luoxuan Weng Xingbo Wang Junyu Lu Yingchaojie Feng Yihan Liu and Wei Chen. 2024. InsightLens: Discovering and exploring insights from conversational contexts in large-language-model-powered data analysis. arXiv:2404.01644 · arXiv
  10. Chen Zhu-Tian Zeyu Xiong Xiaoshuo Yao and Elena Glassman. 2024. Sketch then generate: Providing incremental user feedback and guiding LLM code generation through language-oriented code sketches. arXiv:2405.03998 · arXiv

Metadata

Authors
Xie, L., Zheng, C., Xia, H., Qu, H., & Zhu-Tian, C.
Affiliations
  • University of Minnesota (US)
  • Department of Computer Science and Engineering, University of Minnesota-Twin Cities, United States
Venue
Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
Publisher
ACM
Published
2024-10-11
Pages
1-14
OpenAlex
W4403334501
Cited by
52
References
61
Citations by year
2026: 25 · 2025: 27
FWCI
10.65
Type
conference paper
Language
en
Open access
Locations
  • ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
  • Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
  • arXiv (Cornell University)
Funders
Hong Kong Research Grants Council General Research (16210722)
Topics
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
Keywords
large language models, chatbots, data analysis, LLM-generated code, code visualization, interactive visualization, program comprehension, error detection, user confidence, usability evaluation

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. https://app.cochat.ai/papers/10.1145/3654777.3676374
The paper
Xie, L., Zheng, C., Xia, H., Qu, H., & Zhu-Tian, C. (2024). WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3654777.3676374

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.