Student or educator? Sign up with your school email and get 6 months of Pro free.
CoChat

The DOI matches its Crossref record; 46 of the 46 references it cites were found in a registry or an index; 5 of the 40 sources its claims cite were checked for support.

Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models

Open access
Strobelt, Hendrik, Webson, Albert, Sanh, Victor, Hoover, Benjamin, Beyer, Johanna, Pfister, Hanspeter, Rush, Alexander M. · 2022 · IEEE Transactions on Visualization and Computer Graphics

Abstract

State-of-the-art neural language models can now be used to solve ad-hoc language tasks through zero-shot prompting without the need for supervised training. This approach has gained popularity in recent years, and researchers have demonstrated prompts that achieve strong accuracy on specific NLP tasks. However, finding a prompt for new tasks requires experimentation. Different prompt templates with different wording choices lead to significant accuracy differences. PromptIDE allows users to experiment with prompt variations, visualize prompt performance, and iteratively optimize prompts. We developed a workflow that allows users to first focus on model feedback using small data before moving on to a large data regime that allows empirical grounding of promising prompts using quantitative measures of the task. The tool then allows easy deployment of the newly created ad-hoc models. We demonstrate the utility of PromptIDE (demo: http://prompt.vizhub.ai) and our workflow using several real-world use cases.

Overview

CoChat Summary

TL;DR

PromptIDE is an interactive visual system for zero-shot prompt engineering that lets users create, compare, refine, test, and export prompt templates for ad-hoc NLP tasks. It supports a small-data exploration phase followed by larger-scale quantitative testing, demonstrated on document classification, multiple-choice reading comprehension, and natural language inference.

Key findings

  • PromptIDE automates creation and evaluation of many prompt templates simultaneously and supports different models, tasks, and datasets; it guides users through dataset navigation, prompt variation, prompt refinement, and prompt testing, with export for deployment.
  • In the AG News document-classification use case, replacing the answer choice “Technology” with “Science” after inspecting top-5 ranked predictions “increased performance substantially.”
  • In the RACE multiple-choice use case, “Choose between A, B, C, and D:” consistently gave worse results than other variations; the model often predicted “E” even though it was not an answer choice and tended to predict “C” as the second most frequent prediction, hinting at class imbalance in the underlying model’s training mixture.
  • In the RTE natural language inference use case, T0-3B performance across prompts ranged from 70% to 50% on 20 examples; appending “True or false?” improved performance for all templates, while misleading-extreme prompts outperformed instructive prompts when question marks were present but consistently underperformed when question marks were removed.

Methods

  • Design and system paper: the authors developed PromptIDE through discussions with NLP researchers, multiple design iterations, and internal deployment feedback; they demonstrate utility through several real-world use cases rather than a controlled user study.
  • Data and model: the prototype uses the openly available pre-trained T0-3B large language model through Huggingface, supports standard NLP datasets via Huggingface datasets plus custom CSV/JSON, and demonstrates use cases on AG News, RACE, and RTE.
  • Main analysis: answer options are ranked by decreasing average log-likelihood; the interface visualizes prompt performance with template cards, evaluation chips, stacked bars, confusion matrices, and top-5 ranked generation tokens.

Limitations

  • The paper states that PromptIDE currently supports tasks with a known set of choices, not free-form responses; it also assumes users can deduce how to update prompts from task metrics, and direct advice or gradient-saliency guidance is currently too computationally expensive.
  • The paper reports early feedback criticism that prompt variant variables are limited—e.g., q4 is not available and there is no indicator of maximum prompts allowed—and that some UI elements such as red underlines and technical text were confusing.
  • Evident from the text: evaluation relies on qualitative feedback from two colleagues and use-case demonstrations, not a formal user study; the tool supports only up to three template variables q1, q2, q3.

Model-written summary. Check it against the paper before relying on it.

Integrity signals

Registry

Crossref · 10.1109/tvcg.2022.3209479

Cites

46 of 46 references verified0 citation errors · 0 not checkable

Statistics

Nothing to recompute.

Verification

Claims the paper makes

1 Valid · 13 Not valid · 19 Not checked · 51 Own results and interpretations

24 more claims

Create a free account to see all 84 claims and to verify the 35 references still open.

See all claims — free account
  • Probably not a substantial claim. 46
  • It may state two separate claims as one. 26

Create a free account to see these claims.

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

3 more claims about this paper

Create a free account to see every claim other papers make about this one.

See all claims — free account

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

Each contradiction between two passages listed here was confirmed by an independent check.

1 of 1

Citations

Cites

Show all 39 references

Research works printed but not linked: 9

  1. 0, Github copilot· your ai pair programmer
  2. 10.18653/v1/2021.emnlp-demo.21 · DOI · Paper page
  3. popescu, 2017, Natural language processing meets journalism, Proceedings of the 2017 EMNLP Workshop
  4. radford, 2019, Language models are unsupervised multitask learners, OpenAIRE blog, 1, 9
  5. alammar, 2020, Interfaces for explaining transformer language models
  6. zhidkov, 0, Advanced GPT-3 playground
  7. mayne, 2022, Building games and apps entirely through natural language using openai's code-davinci model, Andrew Mayne Blog
  8. pearce, 2021, What have language models learned?, VISxAI Workshop 2021
  9. 0, OpenAI playground

Cited by

See all 209

Metadata

Authors
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M.
Affiliations
  • IBM (United States) (US)
  • Brown University (US)
  • Hugging Face (US)
  • Harvard University (US)
Venue
IEEE Transactions on Visualization and Computer Graphics
Published
2022
Volume
29
Issue
1
Pages
1-11
ISSN
  • 1077-2626 (ISSN-L)
  • 1941-0506
  • 2160-9306
OpenAlex
W4301393026
Cited by
204
References
39
Citations by year
2026: 21 · 2025: 70 · 2024: 78 · 2023: 35
FWCI
15.06
Type
journal article
Language
en
Open access
Locations
  • IEEE Transactions on Visualization and Computer Graphics (publishedVersion)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion)
  • arXiv (Cornell University)
Funders
National Science Foundation (1901030)
Topics
  • Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
  • Physical Sciences › Computer Science › Computer Graphics and Computer-Aided Design › Data Visualization and Analytics
  • Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
Keywords
prompt engineering, zero-shot prompting, large language models, interactive visualization, prompt optimization, prompt templates, visual analytics

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

The claim verdicts this report counts are experimental: their accuracy is not yet measured.

A language model drew the claims from the paper on 2026-10-04, under version 3 of the claim definition (model deepseek-v4.1-flash, prompt extract-v3).

A second model labelled each claim on 2026-10-04 (model jev-1.13-20260917, definition revision gold-guideline-v0.2).

Cited sources are checked with a language model’s help (qwen3.8-omni-flash).

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review

How to cite

Cite CoChat for the checks on this page, and the paper for its own findings.

This page
CoChat. (2026). Citation check: Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. https://app.cochat.ai/papers/10.1109/tvcg.2022.3209479
The paper
Strobelt, H., Webson, A., Sanh, V., Hoover, B., Beyer, J., Pfister, H., & Rush, A. M. (2022). Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation With Large Language Models. IEEE Transactions on Visualization and Computer Graphics. https://doi.org/10.1109/tvcg.2022.3209479

Data sources

Bibliographic data: Semantic Scholar, OpenAlex, Crossref.

Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).

Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.