WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization
Open accessThe DOI matches its Crossref record; the references it cites are not checked yet.
Abstract
Large language models (LLMs) support data analysis through conversational user interfaces, as exemplified in OpenAI’s ChatGPT (formally known as Advanced Data Analysis or Code Interpreter). Essentially, LLMs produce code for accomplishing diverse analysis tasks. However, presenting raw code can obscure the logic and hinder user verification. To empower users with enhanced comprehension and augmented control over analysis conducted by LLMs, we propose a novel approach to transform LLM-generated code into an interactive visual representation. In the approach, users are provided with a clear, step-by-step visualization of the LLM-generated code in real time, allowing them to understand, verify, and modify individual data operations in the analysis. Our design decisions are informed by a formative study (N=8) probing into user practice and challenges. We further developed a prototype named WaitGPT and conducted a user study (N=12) to evaluate its usability and effectiveness. The findings from the user study reveal that WaitGPT facilitates monitoring and steering of data analysis performed by LLMs, enabling participants to enhance error detection and increase their overall confidence in the results.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“When coupled with validation protocols (e.g., response verification or schema checks in Waitgpt [219]) and observation mechanisms (e.g., feedback loops or audit logs [220, 221]), these boundaries are reinforced, ensuring that each agent not only performs its assigned task but does so transparently and accountably.”
From AI Agents vs. Agentic AI: A Conceptual taxonomy, applications and challenges (2026) · cites this paper as [219]
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
The citation lists are queued
They are fetched later and appear here then.
References 61 · Cited by 52
Research works printed but not linked: 10
- Yida Chen Aoyu Wu Catherine Yeh Trevor DePodesta Kenneth Li Nicholas Castillo Marin Oam Patel Jan Riecke Shivam Raval Olivia Seow Martin Wattenberg and Fernanda Viégas. 2024. Designing a dashboard for transparency and control of conversational AI. arXiv:2406.07882 · arXiv
- Google. 2024. Gemini Advanced: Release updates. Google. Retrieved June 1, 2024 from https://gemini.google.com/updates2024.05.21: updates on data analysis features.
- Yi Guo, Nan Cao, Xiaoyu Qi, Haoyang Li, Danqing Shi, Jing Zhang, Qing Chen, and Daniel Weiskopf. 2023. Urania: Visualizing data analysis pipelines for natural language-based data exploration. arXiv:2306.07760 · arXiv
- Majeed Kazemitabaar, Jack Williams, Ian Drosos, Tovi Grossman, Austin Henley, Carina Negreanu, and Advait Sarkar. 2024. Improving steering and verification in AI-assisted data analysis with interactive task decomposition. arXiv:2407.02651 · arXiv
- Boyan Li Yuyu Luo Chengliang Chai Guoliang Li and Nan Tang. 2024. The dawn of natural language to SQL: Are we fully ready?arXiv:2406.01265 · arXiv
- Hua Shen Tiffany Knearem Reshmi Ghosh Kenan Alkiek Kundan Krishna Yachuan Liu Ziqiao Ma Savvas Petridis Yi-Hao Peng Li Qiwei 2024. Towards bidirectional human-AI alignment: A systematic review for clarifications framework and future directions. arXiv:2406.09264 · arXiv
- Bret Victor. 2011. Up and down the ladder of abstraction: A systematic approach to interactive visualization. Retrieved April 1, 2024 from http://worrydream.com/LadderOfAbstraction/
- Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024. Executable code actions elicit better LLM agents. In Proceedings of the International Conference on Machine Learning (ICML). Article PMLR 235, 13 pages. arXiv:2402.01030 · arXiv
- Luoxuan Weng Xingbo Wang Junyu Lu Yingchaojie Feng Yihan Liu and Wei Chen. 2024. InsightLens: Discovering and exploring insights from conversational contexts in large-language-model-powered data analysis. arXiv:2404.01644 · arXiv
- Chen Zhu-Tian Zeyu Xiong Xiaoshuo Yao and Elena Glassman. 2024. Sketch then generate: Providing incremental user feedback and guiding LLM code generation through language-oriented code sketches. arXiv:2405.03998 · arXiv
Metadata
- Authors
- Xie, L., Zheng, C., Xia, H., Qu, H., & Zhu-Tian, C.
- Affiliations
- University of Minnesota (US)
- Department of Computer Science and Engineering, University of Minnesota-Twin Cities, United States
- Venue
- Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology
- Publisher
- ACM
- Published
- 2024-10-11
- Pages
- 1-14
- OpenAlex
- W4403334501
- Cited by
- 52
- References
- 61
- Citations by year
- 2026: 25 · 2025: 27
- FWCI
- 10.65
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- ACM Symposium on User Interface Software and Technology (UIST) (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by-nc-sa)
- Rare & Special e-Zone (The Hong Kong University of Science and Technology) (submittedVersion)
- arXiv (Cornell University)
- Funders
- Hong Kong Research Grants Council General Research (16210722)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
- Keywords
- large language models, chatbots, data analysis, LLM-generated code, code visualization, interactive visualization, program comprehension, error detection, user confidence, usability evaluation
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. https://app.cochat.ai/papers/10.1145/3654777.3676374
- The paper
- Xie, L., Zheng, C., Xia, H., Qu, H., & Zhu-Tian, C. (2024). WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization. Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology. https://doi.org/10.1145/3654777.3676374
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.