The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 5 of the 40 sources its claims cite were checked for support.
Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models
Open accessAbstract
State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training. This approach has gained popularity in recent years, and researchers have demonstrated prompts that achieve strong accuracy on specific NLP tasks. However, finding a prompt for new tasks requires experimentation. Different prompt templates with different wording choices lead to significant accuracy differences. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts. We developed a workflow that allows users to first focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising prompts using quantitative measures of the task. The tool then allows easy deployment of the newly created ad-hoc models. We demonstrate the utility of PromptIDE (demo: http://prompt.vizhub.ai) and our workflow using several real-world use cases.
Overview
CoChat Summary
TL;DR
PromptIDE is an interactive visual system for zero-shot prompt engineering that lets users create, compare, refine, test, and export prompt templates for ad-hoc NLP tasks. It supports a small-data exploration phase followed by larger-scale quantitative testing, demonstrated on document classification, multiple-choice reading comprehension, and natural language inference.
Key findings
- PromptIDE automates creation and evaluation of many prompt templates simultaneously and supports different models, tasks, and datasets; it guides users through dataset navigation, prompt variation, prompt refinement, and prompt testing, with export for deployment.
- In the AG News document-classification use case, replacing the answer choice “Technology” with “Science” after inspecting top-5 ranked predictions “increased performance substantially.”
- In the RACE multiple-choice use case, “Choose between A, B, C, and D:” consistently gave worse results than other variations; the model often predicted “E” even though it was not an answer choice and tended to predict “C” as the second most frequent prediction, hinting at class imbalance in the underlying model’s training mixture.
- In the RTE natural language inference use case, T0-3B performance across prompts ranged from 70% to 50% on 20 examples; appending “True or false?” improved performance for all templates, while misleading-extreme prompts outperformed instructive prompts when question marks were present but consistently underperformed when question marks were removed.
Methods
- Design and system paper: the authors developed PromptIDE through discussions with NLP researchers, multiple design iterations, and internal deployment feedback; they demonstrate utility through several real-world use cases rather than a controlled user study.
- Data and model: the prototype uses the openly available pre-trained T0-3B large language model through Huggingface, supports standard NLP datasets via Huggingface datasets plus custom CSV/JSON, and demonstrates use cases on AG News, RACE, and RTE.
- Main analysis: answer options are ranked by decreasing average log-likelihood; the interface visualizes prompt performance with template cards, evaluation chips, stacked bars, confusion matrices, and top-5 ranked generation tokens.
Limitations
- The paper states that PromptIDE currently supports tasks with a known set of choices, not free-form responses; it also assumes users can deduce how to update prompts from task metrics, and direct advice or gradient-saliency guidance is currently too computationally expensive.
- The paper reports early feedback criticism that prompt variant variables are limited—e.g., q4 is not available and there is no indicator of maximum prompts allowed—and that some UI elements such as red underlines and technical text were confusing.
- Evident from the text: evaluation relies on qualitative feedback from two colleagues and use-case demonstrations, not a formal user study; the tool supports only up to three template variables q1, q2, q3.
Model-written summary. Check it against the paper before relying on it.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
1 Valid · 13 Not valid · 19 Not checked · 51 Own results and interpretations
At least one cited source doesn’t support this claim or points to a different paper, and none confirmed it — see the verdicts below.
In the paper
“State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training.”
What the sources say
- Disputedcites [3] sourceExperimental — accuracy not yet measured
Our copy supports prompting pretrained LMs to perform tasks without finetuning or added parameters (Abstract; §7 Discussion 'Prompting as an Alternative to Finetuning'), but the claim's decisive qualifier 'zero-shot ... without the need for supervised training' runs contrary to what the paper states: §2.2 and §2.5 describe trigger tokens iteratively updated to maximize label likelihood over batches of *training* data, and the Limitations section states plainly that 'One downside of AUTOPROMPT is that it requires labeled training data.' The paper also restricts itself to masked LMs (BERT BASE, RoBERTa LARGE) on sentiment analysis, NLI, fact retrieval and relation extraction, and only speculates that the method 'can be trivially extended to standard language models ... maybe useful for constructing inputs for models like GPT-3' (§8 Conclusion), so 'state-of-the-art neural language models ... ad-hoc language tasks' is not what this source establishes. (LLM, conf=0.72)
Checked by a reasoning model — may be wrong.
- Not checked yetcites [29] source
- Not checked yetcites [32]
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
- Disputed
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We demonstrate the utility of PromptIDE (demo: http://prompt.vizhub.ai ) and our workflow using several real-world use cases.”
boosted- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper states — it cites no source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“We developed a workflow that allows users to first focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising prompts using quantitative measures of the task.”
plainstates a cause- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“and that it tends to predict 'C' , as it is always the second most frequent prediction when it is not the correct label.”
plainstates an association- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: unclear — It is unclear whether the paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
A result the paper reports from its own data — it cites no other source for it, so there is none to check. The quote shows it is in the paper.
In the paper
“Figure 10(a) shows the output of the selected prompt for some data, and we observe that there is some confusion about the labels.”
plain- Asserted: passes — The authors put it forward as true.
- Checkable: passes — It can be checked against evidence.
- Load-bearing: passes — The paper’s conclusions depend on it.
- Based: passes — The paper gives a reason why it holds.
24 more claims
Create a free account to see all 84 claims and to verify the 35 references still open.
- Probably not a substantial claim. 46
- It may state two separate claims as one. 26
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The cited source doesn’t support the claim as written.
In the citing paper
“Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”
From Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts (2023) · cites this paper as [45]
Checked by a reasoning model — may be wrong.
Experimental — accuracy not yet measuredThe source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“Priming is a powerful way to influence the subsequent responses from LLMs and can contribute to the wide divergence among the interviews (Strobelt et al., 2022).”
From Large Language Models and the Reverse Turing Test (2023) · cites this paper as Strobelt et al., 2022
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“It is well-known that large language models can be highly sensitive to specific prompts (34).”
From Using cognitive psychology to understand GPT-3 (2023) · cites this paper as [34]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Therefore, systems in which the UI did not afford text prompts explicitly made the implicit choice of only using diegetic information as their input to the LLM [6,7,26,41].”
From Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting (2023) · cites this paper as [41]
Experimental — accuracy not yet measuredThe check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Related work showed suggestions automatically after short inactivity [4,6] or gated by a utility function [24].Alternatively, recent work has also explored designs in which users explicitly request suggestions with a hotkey [7,17,26,41,48].”
From Choice Over Control: How Users Write with Large Language Models using Diegetic and Non-Diegetic Prompting (2023) · cites this paper as [41]
Experimental — accuracy not yet measured
3 more claims about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
Each contradiction between two passages listed here was confirmed by an independent check.
Rated minor: it would not touch a result — a value no result rests on, a p-value that stays on the same side of the significance level, or a conflict that is clerical, resolved elsewhere in the paper, or outside its own findings.
In the paper
§ 5.4 Design Iterations and Rationale · p. 7
“The choice for using a templating language (Jinja) over a general programming language (Python) was based on the observation that PromptSource [31] used the same language.”
§ 3 MODEL: PROMPTING FOR NLP · p. 3
“We follow the work of PromptSource [2], where researchers introduced a format for describing prompt templates .”
Citations
Cites
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017) · arXiv · doi:10.48550/arxiv.1706.03762 · Cited by 120,404 · cited 2×
“The idea of the prompting technique has been facilitated by recent improvements in these models, primarily deriving from scaling Transformer neural networks [38].”
Exploiting Generative AI to Scale up Intelligent Tutoring Systems
Jakubův, J., Chvalovský, K., Goertzel, Z., Kaliszyk, C., Olšák, M., Piotrowski, B., Schulz, S., Suda, M., & Urban, J. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.itp.2023.19 · Cited by 76,658
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372 · cited 3×
“Ranking - Compare the rank score of different texts, where f is a function based on p ( y | x ) ( [3], details in Sect.”
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Chain-Of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., & Zhou, D. (2022) · Advances in Neural Information Processing Systems 35 · doi:10.52202/068431-1800 · Cited by 4,505
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2023) · ACM Computing Surveys · doi:10.1145/3560815 · Cited by 4,361 · cited 1×
“For more examples see a recent survey by Liu et al. [21].”
Liu, P., Yuan, W., Fu, J., Jiang, Z., Hayashi, H., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2107.13586 · Cited by 4,352 · cited 1×
“For more examples see a recent survey by Liu et al. [21].”
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022) · arXiv · doi:10.48550/arxiv.2201.11903 · Cited by 4,263 · cited 1×
“Prompting both in the zero-short and few-shot settings has been explored widely in NLP tasks [8,15,32,43].”
Character-level Convolutional Networks for Text Classification
Zhang, X., Zhao, J., & LeCun, Y. (2015) · arXiv (Cornell University) · doi:10.48550/arxiv.1509.01626 · Cited by 3,205 · cited 1×
“The AG News dataset [44] is often used for benchmarking this task and consists of text documents and labels that indicate which topic they are associated with (labels are canonically listed as “World, Sports, Business, Sci/Tech”).”
The Power of Scale for Parameter-Efficient Prompt Tuning
Lester, B., Al-Rfou, R., & Constant, N. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.243 · Cited by 3,096
Research works printed but not linked: 9
- 0, Github copilot· your ai pair programmer
- 10.18653/v1/2021.emnlp-demo.21 · DOI · Paper page
- popescu, 2017, Natural language processing meets journalism, Proceedings of the 2017 EMNLP Workshop
- radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1, 9
- alammar, 2020, Interfaces for explaining transformer language models
- zhidkov, 0, Advanced GPT-3 playground
- mayne, 2022, Building games and apps entirely through natural language using openai's code-davinci model, Andrew Mayne Blog
- pearce, 2021, What have language models learned?, VISxAI Workshop 2021
- 0, OpenAI playground
Cited by
Feuerriegel, S., Hartmann, J., Janiesch, C., & Zschech, P. (2024) · Business & Information Systems Engineering · doi:10.1007/s12599-023-00834-7 · Cited by 2,267
“, 2014) and which entirely new job profiles, such as prompt engineers, may evolve in the near future (Strobelt et al., 2023).”
Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts
Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3581388 · Cited by 1,274
“Prompt design tools to date have focused on supporting professional programmers [45] and NLP practitioners [42], rather than non-AI experts, non-programmers, and other potential end-users of these systems.”
BackgroundAdapted large language models can outperform medical experts in clinical text summarization
Van Veen, D., Van Uden, C., Blankemeier, L., Delbrouck, J.-B., Aali, A., Bluethgen, C., Pareek, A., Polacin, M., Reis, E. P., Seehofnerová, A., et al. (2024) · Nature Medicine · doi:10.1038/s41591-024-02855-5 · Cited by 891
AI-generated feedback on writing: insights into efficacy and ENL student preference
Escalante, J., Pack, A., & Barrett, A. (2023) · International Journal of Educational Technology in Higher Education · doi:10.1186/s41239-023-00425-2 · Cited by 589
“The accuracy and efficacy of GenAI chatbots relies to some extent on prompt engineering (Strobelt et al., 2023), as well as which LLM is used (e.g., BERT, GPT-4).”
MethodUsing cognitive psychology to understand GPT-3
Binz, M., & Schulz, E. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2218523120 · Cited by 559
“It is well-known that large language models can be highly sensitive to specific prompts (34).”
Background
Metadata
- Authors
- Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M.
- Affiliations
- IBM (United States) (US)
- Brown University (US)
- Hugging Face (US)
- Harvard University (US)
- Venue
- IEEE Transactions on Visualization and Computer Graphics
- Published
- 2022
- Volume
- 29
- Issue
- 1
- Pages
- 1-11
- ISSN
- 1077-2626 (ISSN-L)
- 1941-0506
- 2160-9306
- OpenAlex
- W4301393026
- Semantic Scholar
- 0392d58335ce674a70f5e58ac8c438de296a0e6a
- Cited by
- 204
- References
- 39
- Citations by year
- 2026: 21 · 2025: 70 · 2024: 78 · 2023: 35
- FWCI
- 15.06
- Type
- journal article
- Language
- en
- Open access
- green
- arxiv.org
- Licence
- Locations
- IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Funders
- National Science Foundation (1901030)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Keywords
- prompt engineering, zero-shot prompting, large language models, interactive visualization, prompt optimization, prompt templates, visual analytics
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
The claim verdicts this report counts are experimental: their accuracy is not yet measured.
A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).
A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).
Cited sources are checked with a language model’s help (qwen3.8-omni-flash).
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
How to cite
Cite CoChat for the checks on this page, and the paper for its own findings.
- This page
- CoChat. (2026). Citation check: Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. https://app.cochat.ai/papers/10.1109/tvcg.2022.3209479
- The paper
- Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022). Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/tvcg.2022.3209479
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.