Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models
Open accessAbstract
State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training. This approach has gained popularity in recent years, and researchers have demonstrated prompts that achieve strong accuracy on specific NLP tasks. However, finding a prompt for new tasks requires experimentation. Different prompt templates with different wording choices lead to significant accuracy differences. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts. We developed a workflow that allows users to first focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising prompts using quantitative measures of the task. The tool then allows easy deployment of the newly created ad-hoc models. We demonstrate the utility of PromptIDE (demo at http://prompt.vizhub.ai) and our workflow using several real-world use cases.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing LLM interfaces focus only on the input component, not on model parameters or output.
“While our work and these approaches hold the same goal of supporting iteration with LLMs, they focused only on the input component of the generative process-e.g., users cannot test model parameters in PromptIDE [83].”
From Cells, Generators, and Lenses: Design Framework for Object-Oriented Interaction with Large Language Models (2023) · cites this paper as [83]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · see citing works
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · see citing works
Character-level Convolutional Networks for Text ClassificationPreprint
Zhang, X., Zhao, J., & LeCun, Y. (2015) · Neural Information Processing Systems · doi:10.48550/arxiv.1509.01626 · Cited by 7,552 · see citing works
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · see citing works
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1× · see citing works
“For more examples see a recent survey by Liu et al. [21].”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · see citing works
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096 · see citing works
Prefix-Tuning: Optimizing Continuous Prompts for Generation
Li, X. L., & Liang, P. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.353 · Cited by 2,473 · see citing works
The PASCAL Recognising Textual Entailment Challenge
Dagan, I., Glickman, O., & Magnini, B. (2006) · Lecture notes in computer science · doi:10.1007/11736790_9 · Cited by 1,674 · see citing works
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436 · see citing works
Making Pre-trained Language Models Better Few-shot Learners
Gao, T., Fisch, A., & Chen, D. (2021) · Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) · doi:10.18653/v1/2021.acl-long.295 · Cited by 1,380 · see citing works
AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Shin, T., Razeghi, Y., Logan IV, R. L., Wallace, E., & Singh, S. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) · doi:10.18653/v1/2020.emnlp-main.346 · Cited by 1,279 · see citing works
Finetuned Language Models Are Zero-Shot Learners
Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2021) · arXiv · doi:10.48550/arxiv.2109.01652 · Cited by 1,101 · see citing works
RACE: Large-scale ReAding Comprehension Dataset From Examinations
Lai, G., Xie, Q., Liu, H., Yang, Y., & Hovy, E. (2017) · Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d17-1082 · Cited by 1,044 · cited 1× · see citing works
“The RACE dataset [14] is a multiple-choice reading comprehension dataset built from English examinations used for benchmarking this task.”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners
Schick, T., & Schütze, H. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.185 · Cited by 747 · see citing works
Evaluating large language models trained on code
Chen, M. (2024) · TIB Data Manager · doi:10.57702/wbv3e61b · Cited by 737 · see citing works
Multitask Prompted Training Enables Zero-Shot Task Generalization
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. (2021) · arXiv · doi:10.48550/arxiv.2110.08207 · Cited by 612 · see citing works
Self-citationA Multiscale Visualization of Attention in the Transformer Model
Vig, J. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3007 · Cited by 600 · see citing works
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?Preprint
Webson, A., & Pavlick, E. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2109.01247 · Cited by 522 · see citing works
Self-citationCalibrate Before Use: Improving Few-Shot Performance of Language Models
Zhao, T. Z., Wallace, E., Feng, S., Klein, D., & Singh, S. (2021) · arXiv · doi:10.48550/arxiv.2102.09690 · Cited by 513 · see citing works
GLTR: Statistical Detection and Visualization of Generated Text
Gehrmann, S., Strobelt, H., & Rush, A. (2019) · Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/p19-3019 · Cited by 470 · cited 2× · see citing works
“For example, GLTR [9] focuses on visualizing output probability distributions to support humans in detecting whether a provided text was generated by a model.”
Self-citationExplain Yourself! Leveraging Language Models for Commonsense Reasoning
Rajani, N. F., McCann, B., Xiong, C., & Socher, R. (2019) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/p19-1487 · Cited by 421 · see citing works
LSTMVis: A Tool for Visual Analysis of Hidden State Dynamics in Recurrent Neural Networks
Strobelt, H., Gehrmann, S., Pfister, H., & Rush, A. M. (2017) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2017.2744158 · Cited by 399 · see citing works
Self-citationDatasets: A Community Library for Natural Language Processing
Lhoest, Q., Villanova del Moral, A., Jernite, Y., Thakur, A., von Platen, P., Patil, S., Chaumond, J., Drame, M., Plu, J., Tunstall, L., et al. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2021.emnlp-demo.21 · Cited by 343 · cited 1× · see citing works
“It provides access to many standard NLP data using the huggingface datasets [18].”
Self-citationTrue Few-Shot Learning with Language Models
Perez, E., Kiela, D., & Cho, K. (2021) · arXiv · doi:10.48550/arxiv.2105.11447 · Cited by 226 · see citing works
Understanding Hidden Memories of Recurrent Neural Networks
Ming, Y., Cao, S., Zhang, R., Li, Z., Chen, Y., Song, Y., & Qu, H. (2017) · IEEE Conference on Visual Analytics Science and Technology (VAST) · doi:10.1109/vast.2017.8585721 · Cited by 214 · see citing works
Do Prompt-Based Models Really Understand the Meaning of Their Prompts?
Webson, A., & Pavlick, E. (2022) · Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2022.naacl-main.167 · Cited by 203 · see citing works
Self-citationHow many data points is a prompt worth?
Le Scao, T., & Rush, A. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.208 · Cited by 179 · see citing works
Self-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178 · see citing works
Self-citationexBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models
Hoover, B., Strobelt, H., & Gehrmann, S. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2020.acl-demos.22 · Cited by 151 · see citing works
Self-citationTenney, I., Wexler, J., Bastings, J., Bolukbasi, T., Coenen, A., Gehrmann, S., Jiang, E., Pushkarna, M., Radebaugh, C., Reif, E., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.15 · Cited by 138 · see citing works
Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Models
DeRose, J. F., Wang, J., & Berger, M. (2020) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2020.3028976 · Cited by 113 · see citing works
Liu, S., Li, Z., Li, T., Srikumar, V., Pascucci, V., & Bremer, P. (2018) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2018.2865230 · Cited by 48 · see citing works
Dodrio: Exploring Transformer Models with Interactive Visualization
Wang, Z. J., Turko, R., & Chau, D. H. (2021) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2021.acl-demo.16 · Cited by 37 · see citing works
VisQA: X-raying Vision and Language Reasoning in Transformers
Jaunet, T., Kervadec, C., Vuillemot, R., Antipov, G., Baccouche, M., & Wolf, C. (2022) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2021.3114683 · Cited by 30 · see citing works
Progressive Data Science: Potential and ChallengesPreprint
Turkay, Ç., Pezzotti, N., Binnig, C., Strobelt, H., Hammer, B., Keim, D. A., Fekete, J., Palpanas, T., Wang, Y., & Rusu, F. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1812.08032 · Cited by 24 · see citing works
Self-citationPromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S. H., Sanh, V., Yong, Z.-X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · arXiv · doi:10.48550/arxiv.2202.01279 · Cited by 21 · see citing works
Self-citationQASC: A Dataset for Question Answering via Sentence Composition
Khot, T., Clark, P., Guerquin, M., Jansen, P., & Sabharwal, A. (2020) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v34i05.6319 · Cited by 20 · see citing works
T3-Vis: visual analytic for Training and fine-Tuning Transformers in NLP
Li, R., Xiao, W., Wang, L., Jang, H., & Carenini, G. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.26 · Cited by 19 · see citing works
LMdiff: A Visual Diff Tool to Compare Language Models
Strobelt, H., Hoover, B., Satyanaryan, A., & Gehrmann, S. (2021) · Conference on Empirical Methods in Natural Language Processing: System Demonstrations (EMNLP) · doi:10.18653/v1/2021.emnlp-demo.12 · Cited by 16 · see citing works
Self-citationThe FinSim 2020 Shared Task: Learning Semantic Representations for the Financial Domain
Maarouf, I. E., Mansar, Y., Mouilleron, V., & Valsamou-Stanislawski, D. (2020) · Cited by 16 · see citing works
Flagged first, then by citations
Research works printed but not linked: 4
- Q. Lhoest, A. Villanova del Moral, Y. Jernite, A. Thakur, P. von Platen, S. Patil, J. Chaumond, M. Drame, J. Plu, L. Tunstall, J. Davison, M. Šaško, G. Chhablani, B. Malik, S. Brandeis, T. Le Scao, V. Sanh, C. Xu, N. Patry, A. McMillan-Major, P. Schmid, S. Gugger, C. Delangue, T. Matussière, L. Debut, S. Bekman, P. Cistac, T. Goehringer, V. Mustar, F. Lagunas, A. Rush, and T. Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 175-184. Association for Computational Linguistics, Online and Punta Cana, Do- minican Republic, Nov. 2021. doi: 10.18653/v1/2021.emnlp-demo.21 · DOI · Paper page
- A. Pearce. What have language models learned? VISxAI Workshop 2021, https://pair.withgoogle.com/explorables/ fill-in-the-blank/, Jul 2021.
- O. Popescu and C. Strapparava. Natural language processing meets jour- nalism. In Proceedings of the 2017 EMNLP Workshop. Copenhagen, Denmark: Association for Computational Linguistics.-2017, 2017.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
Cited by
Sorted by citations
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 1,409 · see citing works
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 893 · see citing works
Adapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891 · see citing works
AI-generated feedback on writing: insights into efficacy and ENL student preference
Escalante, J., Pack, A., & Barrett, A. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00425-2 · Cited by 589 · see citing works
Using cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559 · see citing works
Prompt engineering in consistency and reliability with the evidence-based guideline for LLMs
Li, W., Chen, X., Deng, X., Wen, H., You, M., Liu, W., Li, Q., & Li, J. (2024) · npj Digital Medicine · doi:10.1038/s41746-024-01029-4 · Cited by 400 · see citing works
Ng, D. T. K., Tan, C. W., & Leung, J. K. L. (2024) · British Journal of Educational Technology · doi:10.1111/bjet.13454 · Cited by 531 · see citing works
Design Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232 · see citing works
Large Language Models and the Reverse Turing Test
Sejnowski, T. J. (2023) · Neural Computation · doi:10.1162/neco_a_01563 · Cited by 169 · see citing works
Kabir, S., Udo-Imeh, D. N., Kou, B., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642596 · Cited by 160 · see citing works
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., WATTENBERG, M. P., & Glassman, E. L. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642016 · Cited by 146 · see citing works
Getting pwn’d by AI: Penetration Testing with Large Language Models
Happe, A., & Cito, J. (2023) · Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering · doi:10.1145/3611643.3613083 · Cited by 144 · see citing works
PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation
Feng, Y., Wang, X., Kam-Kwai, W., Wang, S., Lu, Y., Zhu, M., Wang, B., & Chen, W. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3327168 · Cited by 140 · see citing works
RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressions
Wang, Y., Shen, S., & Lim, B. Y. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581402 · Cited by 120 · see citing works
PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinement
Wang, Z., Huang, Y., Song, D., Ma, L., & Zhang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642803 · Cited by 115 · see citing works
DirectGPT: A Direct Manipulation Interface to Interact with Large Language Models
Masson, D., Malacria, S., Casiez, G., & Vogel, D. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642462 · Cited by 117 · see citing works
Li, L., Fan, L., Atreja, S., & Hemphill, L. (2024) · ACM Transactions on the Web · doi:10.1145/3643829 · Cited by 116 · see citing works
When Software Security Meets Large Language Models: A Survey
Zhu, X., Zhou, W., Han, Q., Ma, W., Wen, S., & Xiang, Y. (2025) · IEEE/CAA Journal of Automatica Sinica · doi:10.1109/jas.2024.124971 · Cited by 105 · see citing works
PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions
Chung, J. J. Y., & Adar, E. (2023) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3586183.3606777 · Cited by 101 · see citing works
Prompting PaLM for Translation: Assessing Strategies and Performance
Vilar, D., Freitag, M., Cherry, C., Luo, J., Ratnakar, V., & Foster, G. F. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.859 · Cited by 125 · see citing works
EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
Kim, T. S., Lee, Y., Shin, J., Kim, Y., & Kim, J. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642216 · Cited by 102 · see citing works
On the Design of AI-powered Code Assistants for Notebooks
McNutt, A. M., Wang, C., DeLine, R. A., & Drucker, S. M. (2023) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580940 · Cited by 81 · see citing works
Dang, H., Goller, S., Lehmann, F., & Buschek, D. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580969 · Cited by 83 · see citing works
Perspectives of Generative AI in Chemistry Education Within the TPACK Framework
Feldman-Maggor, Y., Blonder, R., & Alexandron, G. (2024) · Journal of Science Education and Technology · doi:10.1007/s10956-024-10147-3 · Cited by 100 · see citing works
Generative AIPreprint
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2023) · SSRN Electronic Journal · doi:10.2139/ssrn.4443189 · Cited by 65 · see citing works
Visual cognition in multimodal large language models
Schulze Buschoff, L. M., Akata, E., Bethge, M., & Schulz, E. (2025) · Nature Machine Intelligence · doi:10.1038/s42256-024-00963-y · Cited by 83 · see citing works
Kim, T. S., Lee, Y., Chang, M., & Kim, J. (2023) · Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology · doi:10.1145/3586183.3606833 · Cited by 56 · see citing works
C2Ideas: Supporting Creative Interior Color Design Ideation with a Large Language Model
Hou, Y., Yang, M., Cui, H., Wang, L., Xu, J., & Zeng, W. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642224 · Cited by 59 · see citing works
Sukkar, A. W., Fareed, M. W., Yahia, M. W., Abdalla, S. B., Ibrahim, I., & Senjab, K. (2024) · Buildings · doi:10.3390/buildings14030786 · Cited by 50 · see citing works
Franceschelli, G., & Musolesi, M. (2024) · Journal of Artificial Intelligence Research · doi:10.1613/jair.1.15278 · Cited by 52 · see citing works
Exploring EFL students’ prompt engineering in human–AI story writing: an activity theory perspective
Woo, D. J., Guo, K., & Susanto, H. (2024) · Interactive Learning Environments · doi:10.1080/10494820.2024.2361381 · Cited by 67 · see citing works
A task-oriented framework for generative AI in design
Furtado, L. S., Soares, J. B., & Furtado, V. (2024) · Journal of Creativity · doi:10.1016/j.yjoc.2024.100086 · Cited by 47 · see citing works
Baxter, S. L., Longhurst, C., Millen, M. M., Sitapati, A. M., & Tai‐Seale, M. (2024) · JAMIA Open · doi:10.1093/jamiaopen/ooae028 · Cited by 47 · see citing works
The Silicon Ceiling: Auditing GPT’s Race and Gender Biases in Hiring
Armstrong, L., Liu, A., MacNeil, S., & Metaxa, D. (2024) · ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization (EAAMO) · doi:10.1145/3689904.3694699 · Cited by 59 · see citing works
Visual Analytics for Machine Learning: A Data Perspective Survey
Wang, J., Liu, S., & Zhang, W. Z. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3357065 · Cited by 43 · see citing works
Prompt engineering when using generative AI in nursing education
O’Connor, S., Peltonen, L.-M., Topaz, M., Chen, L.-Y. A., Michalowski, M., Ronquillo, C., Stiglic, G., Chu, C. H., Hui, V., & Denis-Lalonde, D. (2024) · Nurse Education in Practice · doi:10.1016/j.nepr.2023.103825 · Cited by 42 · see citing works
An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks
Bendeck, A., & Stasko, J. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456155 · Cited by 40 · see citing works
RELIC: Investigating Large Language Model Responses using Self-Consistency
Cheng, F., Zouhar, V., Arora, S., Sachan, M., Strobelt, H., & El‐Assady, M. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3641904 · Cited by 40 · see citing works
Self-citationEvaluating large language models for software testing
Li, Y., Liu, P., Wang, H., Chu, J., & Wong, W. E. (2025) · Computer Standards & Interfaces · doi:10.1016/j.csi.2024.103942 · Cited by 42 · see citing works
Fiannaca, A. J., Kulkarni, C. E., Cai, C. J., & Terry, M. A. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3585737 · Cited by 35 · see citing works
Hassan, A. E., Lin, D., Rajbahadur, G. K., Gallaba, K., Cogo, F. R., Chen, B., Zhang, H., Thangarajah, K., Oliva, G. A., Lin, J., et al. (2024) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3663529.3663849 · Cited by 39 · see citing works
PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers’ Workflows
Petridis, S., Terry, M. A., & Cai, C. J. (2024) · Designing Interactive Systems Conference · doi:10.1145/3643834.3661613 · Cited by 40 · see citing works
Interactive Debugging and Steering of Multi-Agent AI Systems
Epperson, W., Bansal, G., Dibia, V. C., Fourney, A., Gerrits, J., Zhu, E. (., & Amershi, S. (2025) · Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3706598.3713581 · Cited by 35 · see citing works
Model-Driven Prompt Engineering
Clarisó, R., & Cabot, J. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models58315.2023.00020 · Cited by 34 · see citing works
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M. A., & Dixon, L. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650755 · Cited by 36 · see citing works
Prompts Are Programs Too! Understanding How Developers Build Software Containing Prompts
Liang, J. T., Lin, M. M., Rao, N., & Myers, B. A. (2025) · Proceedings of the ACM on software engineering. · doi:10.1145/3729342 · Cited by 34 · see citing works
Petridis, S., Wedin, B. D., Wexler, J., Pushkarna, M., Donsbach, A., Goyal, N., Cai, C. J., & Terry, M. (2024) · Proceedings of the 29th International Conference on Intelligent User Interfaces · doi:10.1145/3640543.3645144 · Cited by 31 · see citing works
From Prompt Engineering to Prompt Science with Humans in the Loop
Shah, C. S. (2025) · Communications of the ACM · doi:10.1145/3709599 · Cited by 33 · see citing works
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0 · Cited by 41 · see citing works
CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
Feng, L., Yen, R., You, Y., Fan, M., Zhao, J., & Lu, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642212 · Cited by 29 · see citing works
Gao, L., Lu, J., Shao, Z., Lin, Z., Yue, S., Ieong, C., Sun, Y., Zauner, R. J., Wei, Z., & Chen, S. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456145 · Cited by 33 · see citing works
Kaiser, K. N., Hughes, A. J., Yang, A. D., Turk, A. A., Mohanty, S., Gonzalez, A. A., Patzer, R. E., Bilimoria, K. Y., & Ellis, R. J. (2024) · Journal of Surgical Oncology · doi:10.1002/jso.27821 · Cited by 29 · see citing works
Shahin, M., Chen, F. F., Hosseinzadeh, A., Maghanaki, M., & Eghbalian, A. (2024) · The International Journal of Advanced Manufacturing Technology · doi:10.1007/s00170-024-13167-w · Cited by 28 · see citing works
Enhancing Zero-Shot Crypto Sentiment With Fine-Tuned Language Model and Prompt Engineering
Wahidur, R. S. M., Tashdeed, I., Kaur, M., & Lee, H.-N. (2024) · IEEE Access · doi:10.1109/access.2024.3350638 · Cited by 28 · see citing works
Saleh, Y., Talib, M. W. A., Nasir, Q., & Dakalbab, F. M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1523699 · Cited by 41 · see citing works
Hong, J., Seto, C., Fan, A., & Maciejewski, R. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3536358 · Cited by 25 · see citing works
Gadde, A. (2025) · Journal of Computer Science and Technology Studies · doi:10.32996/jcsts.2025.7.4.66 · Cited by 31 · see citing works
Large-language models facilitate discovery of the molecular signatures regulating sleep and activity
Peng, D., Zheng, L.-B., Liu, D., Han, C., Wang, X., Yang, Y., Song, L., Zhao, M.-Y., Wei, Y.-F., Li, J., et al. (2024) · Nature Communications · doi:10.1038/s41467-024-48005-w · Cited by 29 · see citing works
PrompTHis: Visualizing the Process and Influence of Prompt Editing During Text-to-Image Creation
Guo, Y., Shao, H., Liu, C., Xu, K., & Yuan, X. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3408255 · Cited by 25 · see citing works
Will Code Remain a Relevant User Interface for End-User Programming with Generative AI Models?
Sarkar, A. (2023) · ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software (Onward!) · doi:10.1145/3622758.3622882 · Cited by 25 · see citing works
GesPrompt: Leveraging Co-Speech Gestures to Augment LLM-Based Interaction in Virtual Reality
Hu, X., Ma, D., He, F., Zhu, Z., Hsia, S.-K., Zhu, C., Liu, Z., & Ramani, K. (2025) · Designing Interactive Systems Conference · doi:10.1145/3715336.3735769 · Cited by 25 · see citing works
LLM Comparator: Interactive Analysis of Side-by-Side Evaluation of Large Language Models
Kahng, M., Tenney, I., Pushkarna, M., Liu, M. X., Wexler, J., Reif, E., Kallarackal, K., Chang, M., Terry, M. A., & Dixon, L. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456354 · Cited by 26 · see citing works
Skondras, P., Zervas, P., & Tzimas, G. E. (2023) · Future Internet · doi:10.3390/fi15110363 · Cited by 26 · see citing works
Wang, Q., Erqsous, M., Barner, K. E., & Mauriello, M. L. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3711022 · Cited by 20 · see citing works
Optimizing large vision-language models for context-aware construction safety assessment
Kim, T., Kim, S., Chern, W., Park, S., Kim, D., & Kim, H. (2025) · Automation in Construction · doi:10.1016/j.autcon.2025.106510 · Cited by 21 · see citing works
KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Prompts
Coscia, A. J., & Endert, A. (2023) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2023.3346713 · Cited by 21 · see citing works
Abedi, M., Alshybani, I., Shahadat, M., & Murillo, M. S. (2023) · Qeios · doi:10.32388/md04b0.2 · Cited by 20 · see citing works
Leung, C. H. (2024) · Asian Journal of Contemporary Education · doi:10.55493/5052.v8i2.5101 · Cited by 19 · see citing works
AI in obstetrics: Evaluating residents’ capabilities and interaction strategies with ChatGPT
Desseauve, D., Lescar, R., de la Fournière, B., Ceccaldi, P., & Dziadzko, M. A. (2024) · European Journal of Obstetrics & Gynecology and Reproductive Biology · doi:10.1016/j.ejogrb.2024.09.008 · Cited by 19 · see citing works
Kaiser, K. N., Hughes, A., Yang, A. D., Mohanty, S., Maatman, T., Gonzalez, A. A., Patzer, R. E., Bilimoria, K. Y., & Ellis, R. J. (2025) · Surgery · doi:10.1016/j.surg.2025.109267 · Cited by 19 · see citing works
Promptology: enhancing human-AI interaction in large language models
Olla, P., Elliott, L., Abumeeiz, M., Mihelich, K., & Olson, J. (2024) · Information · doi:10.3390/info15100634 · Cited by 17 · see citing works
Lee, S. Y.-T., Bahukhandi, A., Liu, D., & Ma, K. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456398 · Cited by 18 · see citing works
Development in times of hype: How freelancers explore Generative AI?
Dolata, M., Lange, N., & Schwabe, G. (2024) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3597503.3639111 · Cited by 18 · see citing works
LLM4CVE: Enabling Iterative Automated Vulnerability Repair with Large Language Models
Fakih, M., Dharmaji, R., Bouzidi, H., Araya, G. Q., Ogundare, O., Siddika, M., & Faruque, M. A. A. (2025) · Euromicro Conference on Digital System Design (DSD) · doi:10.1109/dsd67783.2025.00087 · Cited by 18 · see citing works
Explaining Data Patterns in Natural Language with Language Models
Singh, C., Morris, J. X., Aneja, J., Rush, A. M., & Gao, J. (2023) · BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP (BlackboxNLP) · doi:10.18653/v1/2023.blackboxnlp-1.3 · Cited by 15 · see citing works
Self-citationAutomated HEART score determination via ChatGPT: Honing a framework for iterative prompt development
Safranek, C. W., Huang, T. Y., Wright, D. S., Wright, C. X., Socrates, V., Sangal, R. B., Iscoe, M. S., Chartash, D., & Taylor, R. A. (2024) · Journal of the American College of Emergency Physicians Open · doi:10.1002/emp2.13133 · Cited by 15 · see citing works
Appraisal model on how accounting data analytics impacts public sector sustainability reporting
Huy, P. Q., & Phuc, V. K. (2024) · Sustainable Futures · doi:10.1016/j.sftr.2024.100295 · Cited by 15 · see citing works
PromptAid: Visual Prompt Exploration, Perturbation, Testing and Iteration for Large Language Models
Mishra, A., Danzy, B., Soni, U., Arunkumar, A., Huang, J., Kwon, B. C., & Bryan, C. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3535332 · Cited by 62 · see citing works
Generative AI and Prompt Engineering: Transforming Rockburst Prediction in Underground Construction
Kamran, M., Faizan, M., Wang, S., Han, B., & Wang, W. (2025) · Buildings · doi:10.3390/buildings15081281 · Cited by 13 · see citing works
PromptMap: An Alternative Interaction Style for AI-Based Image Generation
Adamkiewicz, K., Woźniak, P. W., Dominiak, J., Romanowski, A., Karolus, J., & Frolov, S. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712150 · Cited by 13 · see citing works
Exploration of applications with ChatGPT for green supply chain management
Zhou, L., Shi, X., Wang, Z., Ma, C., & Gao, L. (2025) · Annals of Operations Research · doi:10.1007/s10479-025-06713-6 · Cited by 15 · see citing works
Review of current and potential uses of large language models in engineering
Nguyen, S. S., & Kittur, J. (2025) · Frontiers in Education · doi:10.3389/feduc.2025.1649650 · Cited by 15 · see citing works
Unmoderated Usability Studies Evolved: Can GPT Ask Useful Follow-up Questions?
Kuric, E., Demcak, P., & Krajcovic, M. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2427978 · Cited by 12 · see citing works
LLMs and Stack Overflow discussions: Reliability, impact, and challenges
Silva, L. D., Samhi, J., & Khomh, F. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2025.112541 · Cited by 22 · see citing works
LLM-Cure: LLM-Based Competitor User Review Analysis for Feature Enhancement
Assi, M., Hassan, S., & Zou, Y. (2025) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3744644 · Cited by 13 · see citing works
Xie, S., Yang, T., Xie, Y., Ying, H., & Wu, Z. (2026) · IEEE Transactions on Fuzzy Systems · doi:10.1109/tfuzz.2026.3665172 · Cited by 10 · see citing works
Artificial intelligence is going to transform the field of endocrinology: an overview
Belkhouribchia, J. (2025) · Frontiers in Endocrinology · doi:10.3389/fendo.2025.1513929 · Cited by 14 · see citing works
JailbreakLens : Visual Analysis of Jailbreak Attacks Against Large Language Models
Feng, Y., Chen, Z., Kang, Z., Wang, S., Tian, H., Zhang, W., Zhu, M., & Chen, W. (2025) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2025.3575694 · Cited by 10 · see citing works
LLM-Agents Driven Automated Simulation Testing and Analysis of small Uncrewed Aerial Systems
Duvvuru, V. S. A., Zhang, B., Vierhauser, M., & Agrawal, A. (2025) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1109/icse55347.2025.00223 · Cited by 10 · see citing works
An AI Chatbot for Explaining Deep Reinforcement Learning Decisions of Service-Oriented Systems
Metzger, A., Bartel, J., & Laufer, J. (2023) · Lecture notes in computer science · doi:10.1007/978-3-031-48421-6_22 · Cited by 9 · see citing works
From Conversational Web to Inclusive Conversations with LLMs
Pucci, E., Piro, L., Andolina, S., & Matera, M. (2024) · International Conference on Advanced Visual Interfaces (AVI) · doi:10.1145/3656650.3656739 · Cited by 9 · see citing works
ModalChorus: Visual Probing and Alignment of Multi-Modal Embeddings via Modal Fusion Map
Ye, Y., Xiao, S., Zeng, X., & Zeng, W. (2024) · IEEE Transactions on Visualization and Computer Graphics · doi:10.1109/tvcg.2024.3456387 · Cited by 9 · see citing works
Schwitter, N. (2025) · Methodological Innovations · doi:10.1177/20597991251313876 · Cited by 9 · see citing works
POEM: Interactive Prompt Optimization for Enhancing Multimodal Reasoning of Large Language Models
He, J., Wang, X., Liu, S., Wu, G., Silva, C. T., & Qu, H. (2025) · IEEE Pacific Visualization Conference (PacificVis) · doi:10.1109/pacificvis64226.2025.00010 · Cited by 14 · see citing works
Nahar, N., Kästner, C., Butler, J., Parnin, C., Zimmermann., T., & Bird, C. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) · doi:10.1109/icse-seip66354.2025.00051 · Cited by 13 · see citing works
Halford, E. (2025) · Policing A Journal of Policy and Practice · doi:10.1093/police/paaf027 · Cited by 11 · see citing works
Mixed-Initiative Interaction with Computational Generative Systems
Lehmann, F. (2023) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3544549.3577061 · Cited by 9 · see citing works
WYTIWYR: A User Intent‐Aware Framework with Multi‐modal Inputs for Visualization Retrieval
Xiao, S., Hou, Y., Jin, C., & Zeng, W. (2023) · Computer Graphics Forum · doi:10.1111/cgf.14832 · Cited by 9 · see citing works
Deep Learning Approaches for Big Data-Driven Metadata Extraction in Online Job Postings
Skondras, P., Ζώτος, Ν., Lagios, D., Zervas, P., Giotopoulos, K. C., & Tzimas, G. E. (2023) · Information · doi:10.3390/info14110585 · Cited by 9 · see citing works
KI-basierte Textkreation im Content Marketing: Design und Evaluation eines effektiven Prompts
Steinmann, N., & Piazza, A. (2024) · HMD Praxis der Wirtschaftsinformatik · doi:10.1365/s40702-024-01058-3 · Cited by 8 · see citing works
- Authors
- Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M.
- Affiliations
- IBM (United States) (US)
- Brown University (US)
- Hugging Face (US)
- Harvard University (US)
- Venue
- arXiv
- Publisher
- arXiv
- Published
- 2022
- Volume
- 29
- Issue
- 1
- Pages
- 1-11
- ISSN
- 1077-2626 (ISSN-L)
- arXiv
- 2208.07852
- OpenAlex
- W4301393026
- Cited by
- 204
- References
- 39
- Citations by year
- 2026: 21 · 2025: 70 · 2024: 78 · 2023: 35
- FWCI
- 15.06
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1901030)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Vision and Pattern Recognition › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt engineering, zero-shot prompting, large language models, interactive visualization, prompt optimization, prompt templates, visual analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
