Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizations
Open accessThe DOI matches its Crossref record; 30 of the 30 references it cites were found in a registry or an index.
Abstract
Natural language interfaces (NLIs) for data visualization are becoming increasingly popular both in academic research and in commercial software. Yet, there is a lack of empirical understanding of how people specify visualizations through natural language. We conducted an online study (N = 102), showing participants a series of visualizations and asking them to provide utterances they would pose to generate the displayed charts. From the responses, we curated a dataset of 893 utterances and characterized the utterances according to (1) their phrasing (e.g., commands, queries, questions) and (2) the information they contained (e.g., chart types, data aggregations). To help guide future research and development, we contribute this utterance dataset and discuss its applications toward the creation and benchmarking of NLIs for visualization.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 2×
“Similarly, the curated utterances can also be used as input to contemporary few-shot learning models such as GPT-3 [6].”
MethodScikit-learn: Machine Learning in Python
(2011) · Journal of Machine Learning Research · doi:10.5555/1953048.2078195 · Cited by 8,539 · cited 1×
“To this aim, we experimented with diferent classifcation techniques including logistic regression, random forests, support vector machines, and Naive Bayes available within Python’s scikitlearn package [19].”
MethodVQA: Visual Question Answering
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., & Parikh, D. (2015) · 2015 IEEE International Conference on Computer Vision (ICCV) · doi:10.1109/iccv.2015.279 · Cited by 4,487
Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1425 · Cited by 1,034
Vega-Lite: A Grammar of Interactive Graphics
Satyanarayan, A., Moritz, D., Wongsuphasawat, K., & Heer, J. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2599030 · Cited by 883 · cited 3×
“Figure 1: Ten visualizations used for the study (all charts were created using Vega-Lite [20]).”
MethodSeq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
Zhong, V., Xiong, C., & Socher, R. (2017) · arXiv (Cornell University) · doi:10.48550/arxiv.1709.00103 · Cited by 733 · cited 2×
“Second, following the methodology used to construct popular NL-to-SQL datasets such as WikiSQL [33], a crowdsourced task could be setup to para-phrase the current set of utterances and then to validate the para-phrased utterances.”
MethodVQA: Visual Question Answering: www.visualqa.org
Agrawal, A., Lu, J., Antol, S., Mitchell, M., Zitnick, C. L., Parikh, D., & Batra, D. (2017) · International Journal of Computer Vision · doi:10.1007/s11263-016-0966-6 · Cited by 570
Das, A., Kottur, S., Gupta, K., Singh, A., Yadav, D., Moura, J. M. F., Parikh, D., & Batra, D. (2017) · 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr.2017.121 · Cited by 445
Formalizing Visualization Design Knowledge as Constraints: Actionable and Extensible Models in Draco
Moritz, D., Wang, C., Nelson, G. L., Lin, H., Smith, A. M., Howe, B., & Heer, J. (2019) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865240 · Cited by 317 · cited 2×
“, chart type), detect attributes in input utterances with toolkits like NL4DV [18], and then determine the most perceptually efective bindings for the classifed chart type’s encoding channels using tools like CompassQL [29] and Draco [17].”
MethodHow Information Visualization Novices Construct Visualizations
Grammel, L., Tory, M., & Storey, M. (2010) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2010.164 · Cited by 264 · cited 1×
“[11] has highlighted how people, especially novices, heavily rely on attributes to specify visualizations (deferring underlying visualization design decisions onto the system).”
BackgroundDataTone: Managing Ambiguity in Natural Language Interfaces for Data Visualization
Gao, T., Dontcheva, M., Adar, E., Liu, Z., & Karahalios, K. G. (2015) · Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology · doi:10.1145/2807442.2807478 · Cited by 242
Eviza: A Natural Language Interface for Visual Analysis
Setlur, V., Battersby, S. E., Tory, M., Gossweiler, R., & Chang, A. X. (2016) · Proceedings of the 29th Annual Symposium on User Interface Software and Technology · doi:10.1145/2984511.2984588 · Cited by 231 · cited 5×
“, [10, 12, 21, 23, 24, 26]), for question answering with charts [13], and for facilitating data-driven communication [7, 14].”
BackgroundMethodInfluentialNarechania, A., Srinivasan, A., & Stasko, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030378 · Cited by 208 · cited 9×
“To benchmark NL4DV’s performance, we configured NL4DV with the three datasets used for the study and executed the 755 singleton”
MethodInfluentialSelf-citationYu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., et al. (2018) · arXiv · doi:10.48550/arxiv.1809.08887 · Cited by 193
Cui, W., Zhang, X., Wang, Y., Huang, H., Chen, B., Fang, L., Zhang, H., Lou, J.-G., & Zhang, D. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934785 · Cited by 153
FlowSense: A Natural Language Interface for Visual Data Exploration within a Dataflow System
Yu, B., & Silva, C. T. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2019.2934668 · Cited by 146 · cited 2×
“, network data exploration [25], dataflow diagram editing [30]) or technical aspects of system development (e.”
BackgroundOrko: Facilitating Multimodal Interaction for Visual Exploration and Analysis of Networks
Srinivasan, A., & Stasko, J. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2745219 · Cited by 140 · cited 1×
“, network data exploration [25], datafow diagram editing [30]) or technical aspects of system development (e.”
BackgroundSelf-citationBeagle: Automated Extraction and Interpretation of Visualizations from the Web
Battle, L., Duan, P., Miranda, Z., Mukusheva, D., Chang, R., & Stonebraker, M. R. (2018) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174168 · Cited by 137 · cited 1×
“Our choice of visualizations was motivated by the popularity of these charts across visualization tools in general [4] as well as within existing NLIs for visualization (e.”
MethodApplying Pragmatics Principles for Interaction with Visual Analytics
Hoque, E., Setlur, V., Tory, M., & Dykeman, I. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744684 · Cited by 132 · cited 2×
“, [10, 12, 21, 23, 24, 26]), for question answering with charts [13], and for facilitating data-driven communication [7, 14].”
BackgroundResultYu, T., Zhang, R., Er, H. Y., Li, S., Xue, E., Pang, B., Lin, X., Tan, Y. C., Shi, T., Li, Z., et al. (2019) · arXiv (Cornell University) · doi:10.48550/arxiv.1909.05378 · Cited by 128
Towards a general-purpose query language for visualization recommendation
Wongsuphasawat, K., Moritz, D., Anand, A., Mackinlay, J. D., Howe, B., & Heer, J. (2016) · Workshop on Human-In-the-Loop Data Analytics (HILDA) · doi:10.1145/2939502.2939506 · Cited by 123 · cited 2×
“, chart type), detect attributes in input utterances with toolkits like NL4DV [18], and then determine the most perceptually efective bindings for the classifed chart type’s encoding channels using tools like CompassQL [29] and Draco [17].”
MethodAnswering Questions about Charts and Generating Visual Explanations
Kim, D. H., Hoque, E., & Agrawala, M. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376467 · Cited by 120 · cited 2×
“, [10, 12, 21, 23, 24, 26]), for question answering with charts [13], and for facilitating data-driven communication [7, 14].”
BackgroundSun, Y., Leigh, J., Johnson, A. E., & Lee, S. (2010) · Lecture notes in computer science · doi:10.1007/978-3-642-13544-6_18 · Cited by 86
InChorus: Designing Consistent Multimodal Interactions for Data Visualization on Tablet Devices
Srinivasan, A., Lee, B., Henry Riche, N., Drucker, S. M., & Hinckley, K. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376782 · Cited by 75 · cited 2×
“Each session included four major phases: (1) introduction & consent , (2) dataset & task description , (3) task trials , and (4) providing prior experience with visualization tools , and consisted of 10 trials corresponding to the 10 visualizations displayed in Figure 1.”
MethodSelf-citationAutomatic Annotation Synchronizing with Textual Description for Visualization
Lai, C., Lin, Z., Jiang, R., Han, Y., Liu, C., & Yuan, X. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376443 · Cited by 72 · cited 1×
“, [10, 12, 21, 23, 24, 26]), for question answering with charts [13], and for facilitating data-driven communication [7, 14].”
BackgroundInferencing underspecified natural language utterances in visual analysis
Setlur, V., Tory, M., & Djalali, A. (2019) · Proceedings of the 24th International Conference on Intelligent User Interfaces · doi:10.1145/3301275.3302270 · Cited by 68 · cited 3×
“, detecting ambiguities [10, 21], inferring underspecified queries [22]).”
BackgroundInfluentialInterweaving Multimodal Interaction With Flexible Unit Visualizations for Data Exploration
Srinivasan, A., Lee, B., & Stasko, J. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.2978050 · Cited by 41
Self-citationTory, M., & Setlur, V. (2019) · 2019 IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast47406.2019.8986918 · Cited by 39 · cited 5×
“Tory and Setlur [28] conducted a wizard-of-oz study to understand people’s expectations of the role of system intelligence in NLIs during visual analysis.”
BackgroundMethodResultInfluentialQuda: Natural Language Queries for Visual Data AnalyticsPreprint
Fu, S., Xiong, K., Ge, X., Tang, S., Chen, W., & Wu, Y. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2005.03257 · Cited by 28 · cited 1×
“[9] collected utterances mapping to Amar et al.”
Understanding the verbal language and structure of end-user descriptions of data visualizations
Metoyer, R. A., Lee, B., Riche, N. H., & Czerwinski, M. (2012) · International Conference on Human Factors in Computing Systems · doi:10.1145/2207676.2208292 · Cited by 19 · cited 1×
“[15] conducted a study where 20 participants worked in pairs with one participant verbally describing a given chart to another participant who attempted to draw the chart based on that description.”
BackgroundSelf-citation
Research works printed but not linked: 2
- Robert Amar, James Eagan, and John Stasko. 2005. Low-level components of analytic activity in information visualization. In IEEE Symposium on Information Visualization. IEEE, Piscataway, NJ, USA, 111–117.
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, Westfield, NJ, USA, 1533–1544.
Cited by
Towards Natural Language Interfaces for Data Visualization: A Survey
Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., & Wang, J. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3148007 · Cited by 181
“[228] made an initial attempt to present a collection of NL utterances with the mapping visualizations, Luo et al.”
BackgroundInfluentialAccessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114770 · Cited by 155
Natural Language to Visualization by Neural Machine Translation
Luo, Y., Tang, N., Li, G., Tang, J., Chai, C., & Qin, X. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114848 · Cited by 149
“Another recent work [48] collected 893 NL queries over three datasets.”
BackgroundChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural Language
Tian, Y., Cui, W., Deng, D., Yi, X., Yang, Y., Zhang, H., & Wu, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3368621 · Cited by 222
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content
Lundgard, A., & Satyanarayan, A. (2022) · DSpace@MIT (Massachusetts Institute of Technology) · Cited by 97
Metadata
- Authors
- Srinivasan, A., Nyapathy, N., Lee, B., Drucker, S. M., & Stasko, J.
- Affiliations
- Tableau Software (United States) (US)
- Georgia Institute of Technology (US)
- Microsoft (United States) (US)
- ORCID
- Drucker, Steven M.
- Venue
- Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems
- Publisher
- ACM
- Published
- 2021-05-06
- Pages
- 1-10
- OpenAlex
- W3128915501
- Semantic Scholar
- fff6b0e394a71edfb077b9a9d28cc6c611c4de6d
- Cited by
- 58
- References
- 23
- Citations by year
- 2026: 10 · 2025: 7 · 2024: 11 · 2023: 12 · 2022: 14 · 2021: 4
- FWCI
- 3.33
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- CHI Conference on Human Factors in Computing Systems (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Artificial Intelligence › Speech and dialogue systems
- Keywords
- natural language interfaces, data visualization, chart types, data aggregation, benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.