Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Natural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language. We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and (2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Similarly, the curated utterances can also be used as input to contemporary few-shot learning models such as GPT-3 [6].”
MethodScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“To this aim, we experimented with diferent classifcation techniques including logistic regression, random forests, support vector machines, and Naive Bayes available within Python’s scikitlearn package [19].”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034
Vega-Lite: A Grammar of Interactive Graphics
Satyanarayan, A., Moritz, D., Wongsuphasawat, K., & Heer, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2599030 · Cited by 883 · cited 3×
“Figure 1: Ten visualizations used for the study (all charts were created using Vega-Lite [20]).”
MethodSeq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
Zhong, V., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00103 · Cited by 733 · cited 2×
“Second, following the methodology used to construct popular NL-to-SQL datasets such as WikiSQL [33], a crowdsourced task could be setup to para-phrase the current set of utterances and then to validate the para-phrased utterances.”
MethodVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco
Moritz, D., Wang, C., Nelson, G. L., Lin, H., Smith, A. M., Howe, B., & Heer, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865240 · Cited by 317 · cited 2×
“, chart type), detect attributes in input utterances with toolkits like NL4DV [18], and then determine the most perceptually efective bindings for the classifed chart type’s encoding channels using tools like CompassQL [29] and Draco [17].”
MethodHow Information Visualization Novices Construct Visualizations
Grammel, L., Tory, M., & Storey, M. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.164 · Cited by 264 · cited 1×
“[11] has highlighted how people, especially novices, heavily rely on attributes to specify visualizations (deferring underlying visualization design decisions onto the system).”
Background
Research works printed but not linked: 2
- Robert Amar, James Eagan, and John Stasko. 2005. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization. IEEE, Piscataway, NJ, USA, 111–117.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, Westfield, NJ, USA, 1533–1544.
Cited by
Towards Natural Language Interfaces for Data Visualization: A Survey
Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., & Wang, J. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3148007 · Cited by 181
“[228] made an initial attempt to present a collection of NL utterances with the mapping visualizations, Luo et al.”
BackgroundInfluentialAccessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155
Natural Language to Visualization by Neural Machine Translation
Luo, Y., Tang, N., Li, G., Tang, J., Chai, C., & Qin, X. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114848 · Cited by 149
“Another recent work [48] collected 893 NL queries over three datasets.”
BackgroundChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural Language
Tian, Y., Cui, W., Deng, D., Yi, X., Yang, Y., Zhang, H., & Wu, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3368621 · Cited by 222
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 97
Metadata
- Authors
- Srinivasan, A., Nyapathy, N., Lee, B., Drucker, S. M., & Stasko, J.
- Affiliations
- Tableau Software (United States) (US)
- Georgia Institute of Technology (US)
- Microsoft (United States) (US)
- ORCID
- Drucker, Steven M.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-10
- OpenAlex
- W3128915501
- Semantic Scholar
- fff6b0e394a71edfb077b9a9d28cc6c611c4de6d
- Cited by
- 58
- References
- 23
- Citations by year
- 2026: 10 · 2025: 7 · 2024: 11 · 2023: 12 · 2022: 14 · 2021: 4
- FWCI
- 3.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language interfaces, data visualization, chart types, data aggregation, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations. https://app.cochat.ai/papers/10.1145/3411764.3445400
- The paper
- Srinivasan, A., Nyapathy, N., Lee, B., Drucker, S. M., & Stasko, J. (2021). Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations. Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3411764.3445400
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.