Conversational Challenges in AI-Powered Data Science: Obstacles, Needs, and Design Opportunities
Open accessAbstract
Large Language Models (LLMs) are being increasingly employed in data science for tasks like data preprocessing and analytics. However, data scientists encounter substantial obstacles when conversing with LLM-powered chatbots and acting on their suggestions and answers. We conducted a mixed-methods study, including contextual observations, semi-structured interviews (n=14), and a survey (n=114), to identify these challenges. Our findings highlight key issues faced by data scientists, including contextual data retrieval, formulating prompts for complex tasks, adapting generated code to local environments, and refining prompts iteratively. Based on these insights, we propose actionable design recommendations, such as data brushing to support context selection, and inquisitive feedback loops to improve communications with AI-based assistants in data-science tools.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Data scientists follow similar interaction patterns to biostatisticians.
“While data scientists follow similar interaction patterns, Chopra et al. [25] found that practitioners spend significant time on prompt preparation and validation”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [25]
Section 4.1.2 (p. 8) reports that participants spent 64% of the measured time preparing prompts, but only the remaining 9%—24 seconds on average—validating code. Section 4.3.3 says validation took the least time of the activities measured, so describing time spent on both preparation and validation as significant overstates the validation finding. The comparison with biostatisticians is framing outside the claim attached to this citation. (LLM, conf=0.84)
Experimental — accuracy not yet measuredPractitioners spend significant time on prompt preparation and validation.
“Chopra et al. [25] found that practitioners spend significant time on prompt preparation and validation”
From Integrating large language models in biostatistical workflows for clinical and translational research (2025) · cites this paper as [25]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · see citing works
Flagged first, then by citations
Research works printed but not linked: 13
- ACM Program. Lang. 7, OOPSLA1, Article 78 (apr 2023), 27 pages. https://doi.org/10.1145/3586030 · DOI · Paper page
- [2]Rohan Bavishi, Caroline Lemieux, Roy Fox, Koushik Sen, and Ion Stoica. 2019. AutoPandas: Neural-Backed Generators for Program Synthesis. Proc. ACM Program. Lang. 3, OOPSLA, Article 168 (oct 2019), 27 pages. https://doi.org/10.1145/3360594 · DOI · Paper page
- [3]Richard A. Becker and William S. Cleveland. 1987. Brushing Scatterplots. Technometrics 29, 2 (1987), 127–142. https://doi.org/10.1080/00401706. · DOI
- 1987.10488204 arXiv:https://www.tandfonline.com/doi/pdf/10.1080/00401706.1987.10488204 · DOI
- [4]Mary Beth Kery and Brad A. Myers. 2017. Exploring exploratory programming. In 2017 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, New York, NY, USA, 25–29. https://doi.org/10.1109/VLHCC.2017.8103446 · DOI · Paper page
- Learn? How Data Scientists Make Sense of Model Behavior. ACM Trans. Comput.-Hum. Interact. 30, 1, Article 1 (mar 2023), 27 pages. https:
- Points, Needs, and Design Opportunities. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376729 · DOI
- Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. arXiv:2107.03374 [cs.LG] · arXiv
- [8]Wenhu Chen. 2023. Large Language Models are few(1)-shot Table Reasoners. arXiv:2210.06710 [cs.CL] · arXiv
- 2023. CoWrangler: Recommender System for Data-Wrangling Scripts. In Companion of the 2023 International Conference on Management of Data (Seattle, WA, USA) (SIGMOD '23). Association for Computing Machinery, New York, NY, USA, 147–150. https://doi.org/10.1145/3555041.3589722 · DOI
- [11]Danyel Fisher, Rob DeLine, Mary Czerwinski, and Steven Drucker. 2012. Interactions with Big Data Analytics. Interactions 19, 3 (may 2012), 50–59. https://doi.org/10.1145/2168931.2168943 · DOI
- [13]Xi Ge and Emerson Murphy-Hill. 2014. Manual Refactoring Changes with Automated Refactoring Validation. In Proceedings of the 36th International Conference on Software Engineering (Hyderabad, India) (ICSE 2014). Association for Computing Machinery, New York, NY, USA, 1095–1105. https://doi.org/10.1145/2568225.2568280 · DOI · Paper page
- [14]Paul Grice. 1991. Studies in the Way of Words. Harvard University Press, Cambridge, Mass. [u.a.].
Cited by
Sorted by citations
Grambow, S. C., Desai, M., Weinfurt, K. P., Lindsell, C. J., Pencina, M. J., Rende, L., & Pomann, G.-M. (2025) · Journal of Clinical and Translational Science · doi:10.1017/cts.2025.10064 · Cited by 7 · see citing works
- Authors
- Chopra, B., Singha, A., Fariha, A., Gulwani, S., Parnin, C., Tiwari, A., & Henley, A. Z.
- Affiliations
- Microsoft (India) (IN)
- ORCID
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2023
- ISSN
- 2331-8422 (ISSN-L)
- arXiv
- 2310.16164
- OpenAlex
- W4387963614
- Cited by
- 16
- References
- 1
- Citations by year
- 2026: 4 · 2025: 8 · 2024: 3 · 2023: 1
- Type
- posted content
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Keywords
- large language models, data science, conversational AI, chatbots, data scientists, prompt engineering, data retrieval, program adaptation, human-AI interaction, data science tools
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
