Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chains
Open accessThe DOI matches its Crossref record; 15 of the 15 references it cites were found in a registry or an index.
Abstract
The emergence of foundation models, such as large language models (LLMs) GPT-4 and text-to-image models DALL-E, has opened up numerous possibilities across various domains. People can now use natural language (i.e., prompts) to communicate with AI to perform tasks. While people can use foundation models through chatbots (e.g., ChatGPT), chat, regardless of the capabilities of the underlying models, is not a production tool for building reusable AI services. APIs like LangChain allow for LLM-based application development but require substantial programming knowledge, thus posing a barrier. To mitigate this, we systematically review, summarise, refine and extend the concept of AI chain by incorporating the best principles and practices that have been accumulated in software engineering for decades into AI chain engineering, to systematize AI chain engineering methodology. We also develop a no-code integrated development environment, Prompt Sapper , which embodies these AI chain engineering principles and patterns naturally in the process of building AI chains, thereby improving the performance and quality of AI chains. With Prompt Sapper, AI chain engineers can compose prompt-based AI services on top of foundation models through chat-based requirement analysis and visual programming. Our user study evaluated and demonstrated the efficiency and correctness of Prompt Sapper.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing
Benjamini, Y., & Hochberg, Y. (1995) · Journal of the Royal Statistical Society Series B: Statistical Methodology · doi:10.1111/j.2517-6161.1995.tb02031.x · Cited by 109,482
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
On the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Reflexion: Language Agents with Verbal Reinforcement Learning
Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., & Yao, S. (2023) · arXiv · doi:10.48550/arxiv.2303.11366 · Cited by 1,087
ProgPrompt: Generating Situated Robot Task Plans using Large Language Models
Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., & Garg, A. (2023) · 2023 IEEE International Conference on Robotics and Automation (ICRA) · doi:10.1109/icra48891.2023.10161317 · Cited by 613 · cited 2×
“Our Sapper IDE is equipped with specialized LLM-based co-pilots to support the magic-enhancing-magic activity in different phases of AI chain engineering.”
BackgroundMethodLarge Language Models Are Human-Level Prompt Engineers
Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022) · arXiv · doi:10.48550/arxiv.2211.01910 · Cited by 330 · cited 1×
“The Design view of our Sapper IDE (see Section 4.1) is equipped with such a requirement elicitation co-pilot that interacts with the engineer to elicit and analyze task requirements.”
MethodInfluentialMeasuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N., & Lewis, M. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.378 · Cited by 303
Measuring and Narrowing the Compositionality Gap in Language Models
Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., & Lewis, M. (2022) · arXiv · doi:10.48550/arxiv.2210.03350 · Cited by 294 · cited 1×
“There are also other strategies to improve prompts gradually, such as self-ask [38], self-reflection [43], and self-consistency [49].”
BackgroundPromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198 · cited 2×
“We conducted three user studies 1 with a total of 30 participants, collectively demonstrating the usefulness, efficiency, correctness, and effectiveness of the proposed tool and its two LLM-based co-pilots in the AI chain generation process.”
BackgroundMethodPromptMaker: Prompt-based Prototyping with Large Language Models
Jiang, E., Olson, K., Toh, E., Molina, A., Donsbach, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3503564 · Cited by 157 · cited 4×
“Jiang et al.[16] develop PromptMaker that enables users to quickly prototype a simple AI service, provide few-shot examples, and test the results.”
BackgroundInfluentialDemystifying Prompts in Language Models via Perplexity Estimation
Gonen, H., Iyer, S., Blevins, T., Smith, N., & Zettlemoyer, L. (2023) · Findings of the Association for Computational Linguistics: EMNLP 2023 · doi:10.18653/v1/2023.findings-emnlp.679 · Cited by 109
Demystifying Prompts in Language Models via Perplexity Estimation
Gonen, H., Iyer, S., Blevins, T., Smith, N. A., & Zettlemoyer, L. (2022) · arXiv · doi:10.48550/arxiv.2212.04037 · Cited by 109
Re3: Generating Longer Stories With Recursive Reprompting and Revision
Yang, K., Tian, Y., Peng, N., & Klein, D. (2022) · arXiv · doi:10.48550/arxiv.2210.06774 · Cited by 91 · cited 3×
“In this section, we review existing AI chain works like [7, 14, 15, 20, 21, 24, 28, 42, 44, 54–56] and define a comprehensive AI chain methodology to systematize the AI chains engineering best principles and practices.”
BackgroundMethodFaithful Reasoning Using Large Language Models
Creswell, A., & Shanahan, M. (2022) · arXiv · doi:10.48550/arxiv.2208.14271 · Cited by 37
Standing on the Shoulders of Giant Frozen Language Models
Levine, Y., Dalmedigos, I., Ram, O., Zeldes, Y., Jannai, D., Muhlgay, D., Osin, Y., Lieber, O., Lenz, B., Shalev-Shwartz, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2204.10019 · Cited by 20
Cited by
Large Language Models for Software Engineering: A Systematic Literature Review
Hou, X., Zhao, Y., Liu, Y., Yang, Z., Wang, K., Li, L., Luo, X., Lo, D., Grundy, J., & Wang, H. (2023) · arXiv · doi:10.48550/arxiv.2308.10620 · Cited by 114
Requirements Engineering Using Generative AI: Prompts and Prompting Patterns
Ronanki, K., Cabrero-Daniel, B., Horkoff, J., & Berger, C. (2024) · Generative AI for Effective Software Development · doi:10.1007/978-3-031-55642-5_5 · Cited by 74
Liu, Y., Lo, S. K., Lu, Q., Zhu, L., Zhao, D., Xu, X., Harrer, S., & Whittle, J. (2025) · Journal of Systems and Software · doi:10.1016/j.jss.2024.112278 · Cited by 51
Self-citationLarge Language Model-Based Agents for Software Engineering: A Survey
Liu, J., Wang, K., Chen, Y., Peng, X., Chen, Z., Zhang, L., & Lou, Y. (2024) · arXiv · doi:10.48550/arxiv.2409.02977 · Cited by 49
Integrating Generative AI in Software Engineering Education: Practical Strategies
Li, Y., Keung, J., & Ma, X. (2024) · 2024 International Symposium on Educational Technology (ISET) · doi:10.1109/iset61814.2024.00019 · Cited by 8
“The proper prompt snapper holds the potential to augment both the efficiency and accuracy of requirement analysis procedures [14].”
Background
Metadata
- Authors
- Cheng, Y., Chen, J., Huang, Q., Xing, Z., Xu, X., & Lu, Q.
- Venue
- ACM Transactions on Software Engineering and Methodology
- Published
- 2024-06-03
- Volume
- 33
- Issue
- 5
- Pages
- 1-24
- ISSN
- 1049-331X (ISSN-L)
- 1557-7392
- OpenAlex
- W4390051886
- Cited by
- 33
- References
- 15
- FWCI
- 7.75
- Type
- journal article
- Language
- en
- Open access
- bronze
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.