The DOI matches its Crossref record; 59 of the 59 references it cites were found in a registry or an index.
Foundation Model Assisted Visual Analytics: Opportunities and Challenges
Open accessAbstract
We explore the integration of foundation models, such as large language models (LLMs) and multimodal LLMs (MLLMs), into visual analytics (VA) systems through intuitive natural language interactions. We survey current research directions in this emerging field, examining how foundation models have already been integrated into key visualisation-related processes in VA: visual mapping, the creation of data visualisations; visualisation observation, the process of generating a finding through visualisation; and visualisation manipulation, changing the viewport or highlighting areas of interest within a visualisation. We also highlight new possibilities that foundation models bring to VA, in particular, the opportunities to use MLLMs to interpret visualisations directly, to integrate multimodal interactions, and to provide guidance to users. We finally conclude with a vision of future VA systems as collaborative partners in analysis and address the prominent challenges in realising this vision through foundation models. Our discussions in this paper aim to guide future researchers working on foundation model assisted VA systems and help them navigate common obstacles when developing these systems.
Overview
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“LLMs are increasingly thought of as an ideal front-end, such as for making dense HR information accessible to employees (75) as exemplified by systems like IBM's AskHR system that serves its large global workforce (76), or for making interaction with data analytics and visualization systems more intuitive and accessible (77).”
From The benefits and dangers of anthropomorphic conversational agents (2025) · cites this paper as [77]
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
LLaMA: Open and Efficient Foundation Language Models
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. (2023) · arXiv · doi:10.48550/arxiv.2302.13971 · Cited by 4,602
Bostock, M., Ogievetsky, V., & Heer, J. (2011) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2011.185 · Cited by 3,165
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Principles of mixed-initiative user interfaces
Horvitz, E. (1999) · Proceedings of the SIGCHI conference on Human factors in computing systems the CHI is the limit - CHI '99 · doi:10.1145/302979.303030 · Cited by 1,490
Gemini: A Family of Highly Capable Multimodal Models
Gemini Team, Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.11805 · Cited by 990
A survey on multimodal large language models
Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., & Chen, E. (2024) · National Science Review · doi:10.1093/nsr/nwae403 · Cited by 893
Knowledge Generation Model for Visual Analytics
Sacha, D., Stoffel, A., Stoffel, F., Kwon, B. C., Ellis, G. P., & Keim, D. A. (2014) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2014.2346481 · Cited by 388
VLAT: Development of a Visualization Literacy Assessment Test
Lee, S., Kim, S.-H., & Kwon, B. C. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598920 · Cited by 292
2020 IEEE Winter Conference on Applications of Computer Vision (WACV)
(2020) · doi:10.1109/wacv45572.2020 · Cited by 277
Maddigan, P., & Susnjak, T. (2023) · IEEE Access · doi:10.1109/access.2023.3274199 · Cited by 242
Narechania, A. A., Srinivasan, A., & Stasko, J. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030378 · Cited by 241
DeepEye: Towards Automatic Data Visualization
Luo, Y., Qin, X., Tang, N., & Li, G. (2018) · IEEE International Conference on Data Engineering (ICDE) · doi:10.1109/icde.2018.00019 · Cited by 239
Characterizing Guidance in Visual Analytics
Ceneda, D., Gschwandtner, T., May, T., Miksch, S., Schulz, H., Streit, M., & Tominski, C. (2016) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2016.2598468 · Cited by 236
Towards Natural Language Interfaces for Data Visualization: A Survey
Shen, L., Shen, E., Luo, Y., Yang, X., Hu, X., Zhang, X., Tai, Z., & Wang, J. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2022.3148007 · Cited by 234
Lee, B., Isenberg, P., Riche, N. H., & Carpendale, S. (2012) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2012.204 · Cited by 211
Reverse‐Engineering Visualizations: Recovering Visual Encodings from Chart Images
Poco, J., & Heer, J. (2017) · Computer Graphics Forum · doi:10.1111/cgf.13193 · Cited by 209
Dibia, V. C. (2023) · Annual Meeting of the Association for Computational Linguistics · doi:10.48550/arxiv.2303.02927 · Cited by 207
Augmenting Visualizations with Interactive Data Facts to Facilitate Interpretation and Communication
Srinivasan, A., Drucker, S. M., Endert, A., & Stasko, J. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865145 · Cited by 203
AI4VIS: Survey on Artificial Intelligence Approaches for Data Visualization
Wu, A., Wang, Y., Shu, X., Moritz, D., Cui, W., Zhang, H., Zhang, D., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3099002 · Cited by 182
Calliope: Automatic Visual Data Story Generation from a Spreadsheet
Shi, D., Xu, X., Sun, F., Shi, Y., & Cao, N. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030403 · Cited by 182
Dibia, V. C. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-demo.11 · Cited by 170
Generative AI for visualization: State of the art and future directions
Ye, Y., Hao, J., Hou, Y., Wang, Z., Xiao, S., Luo, Y., & Zeng, W. (2024) · Visual Informatics · doi:10.1016/j.visinf.2024.04.003 · Cited by 167
ChartGPT: Leveraging LLMs to Generate Charts From Abstract Natural Language
Tian, Y., Cui, W., Deng, D., Yi, X., Yang, Y., Zhang, H., & Wu, Y. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3368621 · Cited by 160
A Survey on ML4VIS: Applying Machine Learning Advances to Data Visualization
Wang, Q., Chen, Z., Wang, Y., & Qu, H. (2021) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3106142 · Cited by 137
Answering Questions about Charts and Generating Visual Explanations
Kim, D. H., Hoque, E., & Agrawala, M. (2020) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376467 · Cited by 133
ChartOCR: Data Extraction from Charts Images via a Deep Hybrid Framework
Luo, J., Li, Z., Wang, J., & Lin, C.-Y. (2021) · IEEE Winter Conference on Applications of Computer Vision · doi:10.1109/wacv48630.2021.00196 · Cited by 119
A Review of Guidance Approaches in Visual Data Analysis: A Multifocal Perspective
Ceneda, D., Gschwandtner, T., & Miksch, S. (2019) · Computer Graphics Forum · doi:10.1111/cgf.13730 · Cited by 77
Foundation models meet visualizations: Challenges and opportunities
Yang, W., Liu, M., Wang, Z., & Liu, S. (2024) · Computational Visual Media · doi:10.1007/s41095-023-0393-x · Cited by 77
Schetinger, V., Bartolomeo, S. D., El‐Assady, M., McNutt, A. M., Miller, M., Passos, J. P. A., & Adams, J. L. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14841 · Cited by 75
LEVA: Using Large Language Models to Enhance Visual Analytics
Zhao, Y., Zhang, Y., Zhang, Y., Zhao, X., Wang, J., Shao, Z., Turkay, Ç., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3368060 · Cited by 65
Chart Question Answering: State of the Art and Future Directions
Hoque, E., Kavehzadeh, P., & Masry, A. A. (2022) · Computer Graphics Forum · doi:10.1111/cgf.14573 · Cited by 62
Oppermann, M., Kincaid, R. H., & Munzner, T. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3030387 · Cited by 62
SciCap: Generating Captions for Scientific Figures
Hsu, T.-Y., Giles, C. L., & Huang, T.-H. (2021) · Findings of the Association for Computational Linguistics: EMNLP 2021 · doi:10.18653/v1/2021.findings-emnlp.277 · Cited by 59
An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks
Bendeck, A., & Stasko, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456155 · Cited by 55
DATATALES: Investigating the use of Large Language Models for Authoring Data-Driven Articles
Sultanum, N., & Srinivasan, A. (2023) · 2023 IEEE Visualization and Visual Analytics (VIS) · doi:10.1109/vis54172.2023.00055 · Cited by 52
Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative Model
Xiao, S., Huang, S., Lin, Y., Ye, Y., & Zeng, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326913 · Cited by 44
Discovering natural language commands in multimodal interfaces
Srinivasan, A., Dontcheva, M., Adar, E., & Walker, S. (2019) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3301275.3302292 · Cited by 42
InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational Notebooks
Lin, Y., Li, H., Yang, L., Wu, A., & Qu, H. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327170 · Cited by 40
Zeng, X., Lin, H., Ye, Y., & Zeng, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456159 · Cited by 39
McCoy, R. T., Yao, S., Friedman, D., Hardy, M., & Griffiths, T. L. (2023) · arXiv · Cited by 37
Large Language Models are Efficient Learners of Noise-Robust Speech RecognitionPreprint
Hu, Y., Chen, C., Yang, C.-H. H., Li, R., Zhang, C., Chen, P., & Chng, E. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.10446 · Cited by 36
Co-adaptive visual data analysis and guidance processes
Sperrle, F., Jeitler, A., Bernard, J., Keim, D. A., & El‐Assady, M. (2021) · Computers & Graphics · doi:10.1016/j.cag.2021.06.016 · Cited by 34
Kim, J., Lee, S., Jeon, H., Lee, K., Bae, H., Kim, B., & Seo, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456215 · Cited by 27
How Good (Or Bad) Are LLMs at Detecting Misleading Visualizations?
Lo, L. Y., & Qu, H. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456333 · Cited by 23
Adaptive user support: Ergonomic design of manually and automatically adaptable software
Oppermann, R. (1994) · CERN Document Server (European Organization for Nuclear Research) · Cited by 23
NL2Color: Refining Color Palettes for Charts with Natural Language
Shi, C., Cui, W., Liu, C., Zheng, C., Zhang, H., Luo, Q., & Ma, X. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3326522 · Cited by 20
The Why and The How: A Survey on Natural Language Interaction in Visualization
Voigt, H., Alaçam, Ö., Meuschke, M., Lawonn, K., & Zarrieß, S. (2022) · Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT) · doi:10.18653/v1/2022.naacl-main.27 · Cited by 14
Natural Language Generation for Visualizations: State of the Art, Challenges and Future Directions
Hoque, E., & Islam, M. S. (2024) · Computer Graphics Forum · doi:10.1111/cgf.15266 · Cited by 13
Ji, Y., & Gao, S. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 10
ChartSense: Interactive Data Extraction from Chart Images
Jung, D., Kim, W., Song, H., Hwang, J.-I., Lee, B., Kim, B., & Seo, J. (2017) · Seoul National University Open Repository (Seoul National University) · Cited by 10
Companion of the 2024 International Conference on Management of Data
(2024) · doi:10.1145/3626246 · Cited by 7
DataTales: Investigating the use of Large Language Models for Authoring Data-Driven ArticlesPreprint
Sultanum, N., & Srinivasan, A. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2308.04076 · Cited by 4
Pay "Attention" to Chart Images for What You Read on Text
Yang, C., Fan, R., Tang, N., Zhang, M., Zhao, X., Fan, J., & Du, X. (2023) · ACM SIGMOD International Conference on Management of Data (SIGMOD) · doi:10.1145/3555041.3589714 · Cited by 3
Probing Language Models' Gesture Understanding for Enhanced Human-AI InteractionPreprint
Wicke, P. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2401.17858 · Cited by 3
Let the Chart Spark: Embedding Semantic Context into Chart with Text-to-Image Generative ModelPreprint
Xiao, S., Huang, S., Lin, Y., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2304.14630 · Cited by 1
InkSight: Leveraging Sketch Interaction for Documenting Chart Findings in Computational NotebooksPreprint
Lin, Y., Li, H., Yang, L., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2307.07922 · Cited by 1
Zeng, X., Lin, H., Ye, Y., et al. (2024) · arXiv (Cornell University) · doi:10.48550/arxiv.2407.20174 · Cited by 1
Motion Editing for Quadruped Characters via Latent Frequency Embedding
Zeng, R., Pan, J., Dai, J. P., Gao, Y., Bai, J., & Qin, H. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3507952 · Cited by 1
Flagged first, then by citations
Research works printed but not linked: 4
- Wenhu Chen, Image from Yang et al. [45]). as direct manipulation, touch, or gesture-based inputs. These modalities allow users to express themselves more precisely, offering greater control and specificity when manipulating visualisations, Direct manipulation techniques which underlie WIMP (Windows, Icons, Menus, Pointer)
- 2005, Illuminating the Path: The Research and Development Agenda for Visual Analytics
- T Gao, 2015, Data-Tone: Managing Ambiguity in Natural Language Interfaces for Data Visualization, Proc. UIST, ACM, 489
- V Setlur, 2016, Eviza: A Natural Language Interface for Visual Analysis, Proc. UIST, ACM, 365
Cited by
Sorted by citations
The benefits and dangers of anthropomorphic conversational agents
Peter, S., Riemer, K., & West, J. D. (2025) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2415898122 · Cited by 110
How do LLMs analyze and interpret data from educational games? A study with GLA experts
Bastos, M., Honda, F., Lima, M. H. T. D. F., Pessoa, M., & Pires, F. (2025) · Anais do ... Simpósio Brasileiro de Informática na Educação/Anais do Simpósio Brasileiro de Informática na Educação · doi:10.5753/sbie.2025.12887 · Cited by 4
InReAcTable: LLM-powered Interactive Visual Data Story Construction from Tabular Data
Aodeng, G., Li, G., Feng, Y., Chen, Q., Zhang, Y., & Liu, C. H. (2025) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3746059.3747719 · Cited by 2
StoryPoint: GenAI-supported domain-specific data story authoring for enterprises
Gunklach, J., Mueller, E., Knaeble, M., & Maedche, A. (2025) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2025.103655 · Cited by 4
Estimating demolition waste from residential interior photos: A Large Language Model solution
Li, Y., Wang, W., Zhao, J., Yang, Y., Zhang, Z., Skibniewski, M. J., & Yuan, J. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106478 · Cited by 4
Metadata
- Authors
- Hutchinson, M., Jianu, R., Slingsby, A., & Madhyastha, P.
- Affiliations
- City, University of London (GB)
- Corresponding author
- Hutchinson, Maeve
- Venue
- Computers & Graphics
- Publisher
- Elsevier BV
- Published
- 2025
- Volume
- 130
- Pages
- 104246-104246
- ISSN
- 0097-8493 (ISSN-L)
- OpenAlex
- W4410544637
- Cited by
- 13
- References
- 58
- Citations by year
- 2026: 7 · 2025: 6
- FWCI
- 4.56
- Type
- journal article
- Language
- en
- Open access
- hybrid
- doi.org
- Licence
- cc-by
- Locations
- Computers & Graphics (publishedVersion, cc-by)
- SSRN Electronic Journal (acceptedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion, cc-by)
- City Research Online (City University London) (submittedVersion)
- arXiv (Cornell University) (cc-by)
- Topics
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Multimodal Machine Learning Applications
- Physical Sciences › Computer Science › Artificial Intelligence › Explainable Artificial Intelligence (XAI)
- Keywords
- foundation models, visual analytics, large language models, multimodal large language models, natural language interaction, visual mapping, multimodal interaction, user guidance, collaborative analysis
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
