Prompting Is Programming: A Query Language for Large Language Models
Open accessAbstract
Large language models have demonstrated outstanding performance on a wide range of tasks such as question answering and code generation. On a high level, given an input, a language model can be used to automatically complete the sequence in a statistically-likely way. Based on this, users prompt these models with language instructions or examples, to implement a variety of downstream tasks. Advanced prompting methods can even imply interaction between the language model, a user, and external tools such as calculators. However, to obtain state-of-the-art performance or adapt language models for specific tasks, complex task- and model-specific programs have to be implemented, which may still require ad-hoc interaction. Based on this, we present the novel idea of Language Model Programming (LMP). LMP generalizes language model prompting from pure text prompts to an intuitive combination of text prompting and scripting. Additionally, LMP allows constraints to be specified over the language model output. This enables easy adaption to many tasks while abstracting language model internals and providing high-level semantics. To enable LMP, we implement LMQL (short for Language Model Query Language), which leverages the constraints and control flow from an LMP prompt to generate an efficient inference procedure that minimizes the number of expensive calls to the underlying language model. We show that LMQL can capture a wide range of state-of-the-art prompting methods in an intuitive way, especially facilitating interactive flows that are challenging to implement with existing high-level APIs. Our evaluation shows that we retain or increase the accuracy on several downstream tasks, while also significantly reducing the required amount of computation or cost in the case of pay-to-use APIs (26-85% cost savings).
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Prompt Cache can automatically convert prompt programs into PML, eliminating the need for manual schema writing.
“To simplify PML writing, Prompt Cache can automatically convert prompt programs (Beurer-Kellner et al., 2023; Guidence, 2023) from languages like Python into PML, eliminating the need for manual schema writing.”
From Prompt Cache: Modular Attention Reuse for Low-Latency Inference (2023) · cites this paper as Beurer-Kellner et al., 2023
Cites
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Transformers: State-of-the-Art Natural Language Processing
Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. (2020) · Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations · doi:10.18653/v1/2020.emnlp-demos.6 · Cited by 8,741 · see citing works
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W., Salakhutdinov, R., & Manning, C. D. (2018) · Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/d18-1259 · Cited by 1,804 · see citing works
Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm
Reynolds, L., & McDonell, K. (2021) · Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411763.3451760 · Cited by 952 · cited 4× · see citing works
“We consider all these works as instances of LMP (also discussed under the term of prompt programming [21, 35]), where the goal is to compose and interact with language models to achieve a specific task.”
BackgroundMethodInfluentialDerivatives of Regular Expressions
Brzozowski, J. (1964) · Journal of the ACM · doi:10.1145/321239.321249 · Cited by 924 · see citing works
PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models
Scholak, T., Schucher, N., & Bahdanau, D. (2021) · Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.779 · Cited by 790 · see citing works
OpenPrompt: An Open-source Framework for Prompt-learning
Ding, N., Hu, S., Zhao, W., Chen, Y., Liu, Z., Zheng, H.-T., & Sun, M. (2022) · Annual Meeting of the Association for Computational Linguistics: System Demonstrations (ACL Demo) · doi:10.18653/v1/2022.acl-demo.10 · Cited by 212 · see citing works
PromptChainer: Chaining Large Language Model Prompts through Visual Programming
Wu, T., Jiang, E., Donsbach, A., Gray, J., Molina, A., Terry, M., & Cai, C. J. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519729 · Cited by 198 · see citing works
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts
Bach, S., Sanh, V., Yong, Z. X., Webson, A., Raffel, C., Nayak, N. V., Sharma, A., Kim, T., Bari, M. S., Fevry, T., et al. (2022) · Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations · doi:10.18653/v1/2022.acl-demo.9 · Cited by 178 · see citing works
Constrained Language Models Yield Few-Shot Semantic Parsers
Shin, R., Lin, C., Thomson, S., Chen, C., Roy, S., Platanios, E. A., Pauls, A., Klein, D., Eisner, J., & Van Durme, B. (2021) · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · doi:10.18653/v1/2021.emnlp-main.608 · Cited by 139 · cited 3× · see citing works
“However, practical methods of constraining LMs in this way [18, 24] still involve a lot of manual implementation effort and model-level understanding of the decoding procedures, tokenization and vocabulary of the LM.”
BackgroundMethodGiving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension
Andor, D., He, L., Lee, K., & Pitler, E. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1609 · Cited by 72 · see citing works
ThinkSum: Probabilistic reasoning over sets using large language models
Ozturkler, B., Malkin, N., Wang, Z., & Jojić, N. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.68 · Cited by 13 · see citing works
PLDI'23 Research Artifacts v0.7 for Programming Large Language Models
Lmql-Lang (2023) · Zenodo (CERN European Organization for Nuclear Research) · doi:10.5281/zenodo.7711342 · Cited by 1 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., WATTENBERG, M. P., & Glassman, E. L. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642016 · Cited by 146 · see citing works
ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
Arawjo, I., Swoopes, C., Vaithilingam, P., Wattenberg, M., & Glassman, E. (2023) · arXiv · doi:10.48550/arxiv.2309.09128 · Cited by 146
Reasoning with Language Model Prompting: A Survey
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., & Chen, H. (2023) · Proceedings of the conference - Association for Computational Linguistics. Meeting · doi:10.18653/v1/2023.acl-long.294 · Cited by 218 · see citing works
Wang, Y., Wang, L., & Siau, K. L. (2024) · International Journal of Human-Computer Interaction · doi:10.1080/10447318.2024.2316376 · Cited by 155 · see citing works
Safeguarding large language models: a survey
Dong, Y., Mu, R., Zhang, Y., Sun, S., Zhang, T., Wu, C., Jin, G., Qi, Y., Hu, J., Meng, J., et al. (2025) · Artificial Intelligence Review · doi:10.1007/s10462-025-11389-2 · Cited by 161 · see citing works
"We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
Liu, M. X., Liu, F., Fiannaca, A. J., Koo, T., Dixon, L., Terry, M. A., & Cai, C. J. (2024) · CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA) · doi:10.1145/3613905.3650756 · Cited by 80 · see citing works
Leveraging LLMs for the Quality Assurance of Software Requirements
Lubos, S., Felfernig, A., Tran, T. N. T., Garber, D., Mansi, M. E., Erdeniz, S. P., & Le, V.-M. (2024) · IEEE International Requirements Engineering Conference (RE) · doi:10.1109/re59067.2024.00046 · Cited by 72 · see citing works
Generative AI in the Wild: Prospects, Challenges, and Strategies
Sun, Y., Jang, E., Ma, F., & Wang, T. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642160 · Cited by 64 · see citing works
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., Vardhamanan, S., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., et al. (2023) · arXiv · doi:10.48550/arxiv.2310.03714 · Cited by 60 · see citing works
LLM-generated Explanations for Recommender Systems
Lubos, S., Tran, T. N. T., Felfernig, A., Erdeniz, S. P., & Le, V.-M. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3665185 · Cited by 62 · see citing works
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R., Ribeiro, A., Robinson, C., Loeb, S., & Demszky, D. (2024) · doi:10.21203/rs.3.rs-5363154/v1 · Cited by 53 · see citing works
Automatic structuring of radiology reports with on-premise open-source large language models
Woźnicki, P., Laqua, C., Fiku, I., Hekalo, A., Truhn, D., Engelhardt, S., Kather, J. N., Foersch, S., D’Antonoli, T. A., dos Santos, D. P., et al. (2024) · European Radiology · doi:10.1007/s00330-024-11074-y · Cited by 47 · see citing works
A unified industrial large knowledge model framework in Industry 4.0 and smart manufacturing
Lee, J., & Su, H. (2024) · International Journal of AI for Materials and Design · doi:10.36922/ijamd.3681 · Cited by 45 · see citing works
Towards AI-Assisted Synthesis of Verified Dafny Methods
Misu, M. R. H., Lopes, C. V., Ma, I., & Noble, J. (2024) · Proceedings of the ACM on Software Engineering · doi:10.1145/3643763 · Cited by 38 · see citing works
Wang, Y., Guo, S., & Tan, C. W. (2025) · IEEE Software · doi:10.1109/ms.2025.3549628 · Cited by 38 · see citing works
Agents in software engineering: survey, landscape, and vision
Wang, Y., Zhong, W., Huang, Y., Shi, E., Yang, M., Chen, J., Li, H., Ma, Y., Wang, Q., & Zheng, Z. (2025) · Automated Software Engineering · doi:10.1007/s10515-025-00544-2 · Cited by 50 · see citing works
Model-Driven Prompt Engineering
Clarisó, R., & Cabot, J. (2023) · ACM/IEEE International Conference on Model Driven Engineering Languages and Systems (MODELS) · doi:10.1109/models58315.2023.00020 · Cited by 34 · see citing works
Galli, C., Gavrilova, A. V., & Calciolari, E. (2025) · Information · doi:10.3390/info16050378 · Cited by 36 · see citing works
CoPrompt: Supporting Prompt Sharing and Referring in Collaborative Natural Language Programming
Feng, L., Yen, R., You, Y., Fan, M., Zhao, J., & Lu, Z. (2024) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642212 · Cited by 29 · see citing works
Efficient Guided Generation for Large Language Models
Willard, B. T., & Louf, R. (2023) · arXiv · doi:10.48550/arxiv.2307.09702 · Cited by 27 · see citing works
Saleh, Y., Talib, M. W. A., Nasir, Q., & Dakalbab, F. M. (2025) · Frontiers in Computer Science · doi:10.3389/fcomp.2025.1523699 · Cited by 41 · see citing works
Automated Program Refinement: Guide and Verify Code Large Language Model with Refinement Calculus
Cai, Y., Hóu, Z., Sanán, D., Luan, X., Lin, Y., Sun, J., & Dong, J. S. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3704905 · Cited by 27 · see citing works
How Effective are Large Language Models in Generating Software Specifications?
Xie, D., Yoo, B., Jiang, N., Kim, M., Tan, L., Zhang, X., & Lee, J. S. (2025) · IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) · doi:10.1109/saner64311.2025.00014 · Cited by 47 · see citing works
CoLadder: Manipulating Code Generation via Multi-Level Blocks
Yen, R., Zhu, J., Suh, S., Xia, H., & Zhao, J. (2024) · ACM Symposium on User Interface Software and Technology (UIST) · doi:10.1145/3654777.3676357 · Cited by 22 · see citing works
Pereira, A. F., & Mello, R. F. (2025) · IEEE Access · doi:10.1109/access.2025.3584060 · Cited by 22 · see citing works
Chun, J., & Elkins, K. (2023) · International Journal of Digital Humanities · doi:10.1007/s42803-023-00069-8 · Cited by 19 · see citing works
ECHO: An Approach to Enhance Use Case Quality Exploiting Large Language Models
Vito, G. D., Palomba, F., Gravino, C., Martino, S. D., & Ferrucci, F. (2023) · Euromicro Conference on Software Engineering and Advanced Applications (SEAA) · doi:10.1109/seaa60479.2023.00017 · Cited by 17 · see citing works
Hristova, T., Magee, L., & Soldatić, K. (2024) · AI & Society · doi:10.1007/s00146-024-02039-2 · Cited by 16 · see citing works
Investigating translation for Indic languages with BLOOMZ-3b through prompting and LoRA fine-tuning
Nair, A. R., Gupta, D., & Premjith, B. (2024) · Scientific Reports · doi:10.1038/s41598-024-74617-9 · Cited by 16 · see citing works
Impromptu: a framework for model-driven prompt engineering
Morales, S., Clarisó, R., & Cabot, J. (2025) · Software & Systems Modeling · doi:10.1007/s10270-024-01235-4 · Cited by 16 · see citing works
Prompt Cache: Modular Attention Reuse for Low-Latency Inference
Gim, I., Chen, G., Lee, S.-S., Sarda, N., Khandelwal, A., & Zhong, L. (2023) · arXiv · doi:10.48550/arxiv.2311.04934 · Cited by 18 · see citing works
Type-Constrained Code Generation with Language Models
Mündler-Sasahara, N., He, J.-X., Wang, H., Sen, K., Song, D., & Vechev, M. T. (2025) · Proc. ACM Program. Lang. · doi:10.1145/3729274 · Cited by 77 · see citing works
Self-citationDiyab, A., Frost, R. M., Fedoruk, B. D., & Diyab, A. (2025) · Education Sciences · doi:10.3390/educsci15020156 · Cited by 13 · see citing works
SGLang: Efficient Execution of Structured Language Model Programs
Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., et al. (2023) · arXiv · doi:10.48550/arxiv.2312.07104 · Cited by 145 · see citing works
“We refer to these programs as "Language Model Programs" (LM Programs) [4, 20].”
InfluentialEldem, A., & Eldem, H. (2026) · Scientific Reports · doi:10.1038/s41598-026-35003-9 · Cited by 12 · see citing works
AdaptiveGPT: Towards Intelligent Adaptive Learning
Sachete, A. D. S., Loiola, A. V. D. S. ́. D. F., & Gomes, R. S. (2024) · Multimedia Tools and Applications · doi:10.1007/s11042-024-20144-8 · Cited by 12 · see citing works
How to Write Effective Prompts for Screening Biomedical Literature Using Large Language Models
Colangelo, M. T., Guizzardi, S., Meleti, M., Calciolari, E., & Galli, C. (2025) · BioMedInformatics · doi:10.3390/biomedinformatics5010015 · Cited by 11 · see citing works
Semantic Integrity Constraints: Declarative Guardrails for AI-Augmented Data Processing Systems
Lee, A. W., Chan, J., Fu, M. C., Kim, N. H.-Y., Mehta, A., Raghavan, D., & Çetintemel, U. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3749646.3749677 · Cited by 9 · see citing works
Code Detection for Hardware Acceleration Using Large Language Models
Martínez, P. A., Bernabé, G., & García, J. M. (2024) · IEEE Access · doi:10.1109/access.2024.3372853 · Cited by 10 · see citing works
Over the Rainbow: How to Fly over with ChatGPT in Tourism
Kim, T.-K. (2023) · Journal of Smart Tourism · doi:10.52255/smarttourism.2023.3.1.5 · Cited by 9 · see citing works
Meta-Harness: End-to-End Optimization of Model Harnesses
Lee, Y., Nair, R., Zhang, Q., Lee, K., Khattab, O., & Finn, C. (2026) · arXiv · doi:10.48550/arxiv.2603.28052 · Cited by 3 · see citing works
“LMQL [8], LangChain [13], and DSPy [23] make prompt engineering more systematic by providing higher-level interfaces for prompt templates, control flow, and modular LLM pipelines.”
NLDesign: A UI Design Tool for Natural Language Interfaces
Zhang, T., Fu, P., Liu, J., Zhang, Y., & Chen, X. (2024) · ACM Turing Celebration Conference - China (ACM TURC) · doi:10.1145/3674399.3674455 · Cited by 8 · see citing works
Νικολακόπουλος, Α., Λίτκε, Α., Psychas, A., Veroni, E., & Varvarigou, T. A. (2025) · IEEE Access · doi:10.1109/access.2025.3556973 · Cited by 10 · see citing works
NeSyCoCo: A Neuro-Symbolic Concept Composer for Compositional Generalization
Kamali, D., Barezi, E. J., & Kordjamshidi, P. (2025) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v39i4.32439 · Cited by 8 · see citing works
Hassan, A. (2026) · Education and Information Technologies · doi:10.1007/s10639-026-13946-w · Cited by 22 · see citing works
MeshAgent: enabling reliable network management with large language models
Zhou, Y., Hsieh, K., Mani, S. K., Kandula, S., & Liu, Z. (2025) · Proceedings of the ACM on Measurement and Analysis of Computing Systems · doi:10.1145/3771567 · Cited by 7 · see citing works
An Emerging Design Space of How Tools Support Collaborations in AI Design and Development
Muralikumar, M. D., & McDonald, D. W. (2025) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3701181 · Cited by 7 · see citing works
Wang, X., Zhong, W., Huang, K., & Liang, B. (2025) · International Journal of Information Management · doi:10.1016/j.ijinfomgt.2025.103009 · Cited by 7 · see citing works
AskIt: Unified Programming Interface for Programming with Large Language Models
Okuda, K., & Amarasinghe, S. P. (2024) · Proceedings of the CGO/Proceedings of the ... CGO · doi:10.1109/cgo57630.2024.10444830 · Cited by 7 · see citing works
Relational Programming with Foundational Models
Li, Z., Huang, J., Liu, J., Zhu, F., Zhao, E., Dodds, W., Velingker, N., Rajeev Alur, A., & Naik, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i9.28934 · Cited by 6 · see citing works
Complex QA and language models hybrid architectures, SurveyPreprint
Daull, X., Bellot, P., Bruno, E., Martin, V., & Murisasco, É. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2302.09051 · Cited by 17 · see citing works
Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise
Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024) · arXiv · doi:10.48550/arxiv.2410.03017 · Cited by 6 · see citing works
“This observation underscores the critical role of the human in guiding the AI to provide the most contextually appropriate and impactful instructional support.”
Towards Evaluating Large Language Models for Graph Query Generation
Munir, S., & Aldini, A. (2025) · Communications in computer and information science · doi:10.1007/978-3-031-90341-0_3 · Cited by 5 · see citing works
Developing Conversational Intelligent Tutoring for Speaking Skills in Second Language Learning
Lee, J., Huang, J., Cho, M., Roh, Y., Kwon, O., & Lee, Y. (2024) · Lecture notes in computer science · doi:10.1007/978-3-031-63028-6_11 · Cited by 5 · see citing works
TAKAGI, H., Moriya, S., Sato, T., Nagao, M., & Higuchi, K. (2025) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3708359.3712119 · Cited by 5 · see citing works
Wencker, T., Borst, J., & Niekler, A. (2025) · Evaluation · doi:10.1177/13563890251330911 · Cited by 5 · see citing works
Automatically Testing Functional Properties of Code Translation Models
Enişer, H. F., Wüstholz, V., & Christakis, M. (2024) · Proceedings of the AAAI Conference on Artificial Intelligence · doi:10.1609/aaai.v38i19.30097 · Cited by 4 · see citing works
JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Models
Geng, S., Cooper, H., Moskal, M., Jenkins, S., Berman, J., Ranchin, N., West, R., Horvitz, E., & Nori, H. (2025) · arXiv · doi:10.48550/arxiv.2501.10868 · Cited by 3 · see citing works
“Recently, highly optimized grammar-constrained decoding frameworks [Guidance AI, 2023; Beurer-Kellner et al., 2023; Willard & Louf, 2023; Kuchnik et al., 2023; Zheng et al., 2024; Dong et al., 2024] have been developed to improve the efficiency and usability of constrained decoding.”
Berro, A., dos Santos, V. G., Benatallah, B., & Benabdeslem, K. (2025) · Lecture notes in computer science · doi:10.1007/978-3-031-94569-4_9 · Cited by 3 · see citing works
VAP3: Variation-Aware Prompt Performance Prediction
Arabzadeh, N., & Ebrahim Bagheri, A. (2025) · International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR) · doi:10.1145/3726302.3730264 · Cited by 3 · see citing works
Agua: A Concept-Based Explainer for Learning-Enabled Systems
Patel, S., Han, D., Narodytska, N., & Jyothi, S. A. (2025) · ACM SIGCOMM Conference (SIGCOMM) · doi:10.1145/3718958.3754341 · Cited by 3 · see citing works
GreenFoodLens: Sustainability Labels for Food Recommendation
Balloccu, G., Boratto, L., Fenu, G., Marras, M., Medda, G., & Murgia, G. (2025) · ACM Conference on Recommender Systems (RecSys) · doi:10.1145/3705328.3748165 · Cited by 3 · see citing works
Self-Directed Learning for Community Health Workers in Malawi Through Generative AI
Taylor, A., Magwira, M., Chamangwana, C., Chapuma, E., Liwewe, T., & Kankhwali, C. (2024) · IEEE International Conference on Healthcare Informatics (ICHI) · doi:10.1109/ichi61247.2024.00092 · Cited by 3 · see citing works
Chelliah, S. K., Arivazhagan, N., Alfarraj, O., & Tolba, A. K. R. (2025) · Journal of Circuits Systems and Computers · doi:10.1142/s0218126625502342 · Cited by 3 · see citing works
Kani: A Lightweight and Highly Hackable Framework for Building Language Model Applications
Zhu, A., Dugan, L. D., Hwang, A., & Callison-Burch, C. (2023) · Workshop for NLP Open Source Software (NLP-OSS) · doi:10.18653/v1/2023.nlposs-1.8 · Cited by 3 · see citing works
LLM-Guided Multi-Agent System for Natural Language-Based Robot Navigation
Samarathunga, K., Gurusinghe, R., Sivasothynathan, K., Wanigasekara, C., Mars, J., & Velmanickam, L. (2025) · IEEE World AI IoT Congress (AIIoT) · doi:10.1109/aiiot65859.2025.11105295 · Cited by 2 · see citing works
Optimized Batch Prompting for Cost-Effective LLMs
Ji, Z., Wang, X., Luo, Z., Xie, Z., & Zhang, M. (2025) · Proceedings of the VLDB Endowment · doi:10.14778/3734839.3734853 · Cited by 2 · see citing works
Al-Turki, D., Basurra, S. S., Gaber, M. M., Attasi, B., & Abdelsamea, M. M. (2025) · International Joint Conference on Neural Networks (IJCNN) · doi:10.1109/ijcnn64981.2025.11227910 · Cited by 2 · see citing works
Evaluating and improving LLM-based competitive program generation
Wei, M., Li, Z., Chen, X., Zheng, M., Qu, Z., Yu, C., Chen, S. C. J., & Ju, X. (2025) · Information and Software Technology · doi:10.1016/j.infsof.2025.107977 · Cited by 2 · see citing works
Huang, L., Liu, Z.-G., Yu, C., Zhu, T.-L., & Yan, B. (2026) · Italian National Conference on Sensors · doi:10.3390/s26062006 · Cited by 2 · see citing works
Improving Recommender Systems with Large Language Models
Lubos, S. (2024) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3631700.3664919 · Cited by 3 · see citing works
Generating Function Names to Improve Comprehension of Synthesized Programs
Nazari, A., Swayamdipta, S., Chattopadhyay, S., & Raghothaman, M. (2024) · IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC) · doi:10.1109/vl/hcc60511.2024.00035 · Cited by 2 · see citing works
Opportunistically Parallel Lambda Calculus
Mell, S., Καλλάς, Κ., Zdancewic, S., & Bastani, O. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763143 · Cited by 2 · see citing works
DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
Singhvi, A., Shetty, M., Tan, S., Potts, C., Sen, K., Zaharia, M., & Khattab, O. (2023) · arXiv · doi:10.48550/arxiv.2312.13382 · Cited by 2 · see citing works
McEval: Massively Multilingual Code Evaluation
Chai, L., Liu, S., Yang, J., Yin, Y., Jin, K., Liu, J., Sun, T., Zhang, G., Ren, C., Guo, H., et al. (2024) · arXiv · doi:10.48550/arxiv.2406.07436 · Cited by 1 · see citing works
From Prompts to Contracts: Harness Engineering for Auditable Enterprise LLM Agents
Ahn, J., & Kim, M. (2026) · arXiv · doi:10.48550/arxiv.2607.08028 · Cited by 1 · see citing works
“At the call level, the Language Model Query Language (LMQL) constrains model decoding with declarative constraints [12], Instructor enforces typed output schemas with automatic validation retries [41], and Guidance interleaves control flow with constrained generation within a single model session…”
Üretken uzamsal zekânın getirdiği paradigma değişimi
Güney, C. (2024) · Journal of Geodesy and Geoinformation · doi:10.9733/jgg.2024r0009.t · Cited by 1 · see citing works
Membership Testing for Semantic Regular Expressions
Huang, Y., Amini, M., Glaunec, A. L., Mamouras, K., & Raghothaman, M. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3729300 · Cited by 1 · see citing works
Zhao, X., & Jebelli, H. (2025) · ASCE International Conference on Computing in Civil Engineering (i3CE) · doi:10.1061/9780784486115.099 · Cited by 1 · see citing works
Revisiting the role of linguistic knowledge in large language models through prompting
Sataer, Y., Gao, M., Shi, C., Li, B., & Gao, Z. (2026) · Knowledge-Based Systems · doi:10.1016/j.knosys.2026.115667 · Cited by 1 · see citing works
Njuguna, G. N., & Min, Q. (2026) · AI & Society · doi:10.1007/s00146-026-02973-3 · Cited by 1 · see citing works
Shoemaker, T. (2026) · AI & Society · doi:10.1007/s00146-026-02897-y · Cited by 1 · see citing works
Guo, Y., Liu, G., Zhu, J., Chu, H., Wen, S., & Wang, F. (2026) · Biomimetic Intelligence and Robotics · doi:10.1016/j.birob.2026.100305 · Cited by 1 · see citing works
Jamal, S., Ahmad, A., & ZESHAN, D. F. (2026) · International Journal of Software Engineering and Knowledge Engineering · doi:10.1142/s0218194026300010 · Cited by 1 · see citing works
Debnath, T., Siddiky, M. N. A., Rahman, M. E., Das, P., Guha, A. K., Rahman, M. R., & Kabir, H. M. D. (2026) · Computer Science Review · doi:10.1016/j.cosrev.2026.100979 · Cited by 2 · see citing works
Large language models for interpretation of health checkup results
You, J., & Shin, H. (2026) · npj Digital Medicine · doi:10.1038/s41746-026-02467-y · Cited by 1 · see citing works
TraitsPrompt: Does Personality Traits Influence the Performance of a Large Language Model?
Zhu, Q., Lan, T., Xue, X., Yu, Z., & He, H. (2024) · International Conference on Computer Supported Cooperative Work in Design (CSCWD) · doi:10.1109/cscwd61410.2024.10580436 · Cited by 1 · see citing works
Some Useful Things to Know When Combining IR and NLP: The Easy, the Hard and the Ugly
Alonso, O., & Church, K. (2024) · ACM International Conference on Web Search and Data Mining (WSDM) · doi:10.1145/3616855.3636452
Inteligencia Artificial y sus subcampos de estudio: una revisión histórica
Agreda, Z. Y. S. (2024) · Revista peruana de computación y sistemas · doi:10.15381/rpcs.v6i2.28857
Saputra, I. D., Harahap, N. S., Agustian, S., Fikry, M., & Oktavia, L. (2024) · Journal of Computer System and Informatics (JoSYC) · doi:10.47065/josyc.v6i1.6182
Gim, I., & Zhong, L. (2025) · Workshop on Hot Topics in Operating Systems · doi:10.1145/3713082.3730398
SnipGen: A Mining Repository Framework for Evaluating LLMs for Code
Rodríguez-Cárdenas, D., Velasco, A., & Poshyvanyk, D. (2025) · International Conference on Mining Software Repositories (MSR) · doi:10.1109/msr66628.2025.00084
Leveraging LLMs to Explain the Consequences of Recommendations
Lubos, S., Gartner, M., Felfernig, A., & Willfort, R. (2025) · ACM Conference on User Modeling, Adaptation and Personalization (UMAP) · doi:10.1145/3699682.3728328
Polymer: Development Workflows as Software
Parthasarathy, D., Yu, Y., & Barr, E. T. (2025) · ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) · doi:10.1145/3696630.3728494
Trustworthy urban digital twin: a RAG-based architecture for integrating verifiable knowledge
Xie, X., Herrera, M., Tang, X.-Y., James, P., & Kassem, M. (2026) · IET conference proceedings. · doi:10.1049/icp.2026.0487
Efficient Function Orchestration for Large Language Models
Liu, X., Di, P., Li, C., Sun, J., & Wang, J. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3619112 · Cited by 1 · see citing works
Composable Effect Handling for Programming LLM-Integrated Scripts
Wang, D. (2025) · ACM SIGPLAN International Workshop on Language Models and Programming Languages (LMPL) · doi:10.1145/3759425.3763396
MTP: A Meaning-Typed Language Abstraction for AI-Integrated Programming
Dantanarayana, J. L., Kang, Y., Sivasothynathan, K., Clarke, C., Li, B., Kashmira, S., Flautner, K., Tang, L., & Mars, J. (2025) · Proceedings of the ACM on Programming Languages · doi:10.1145/3763092
Prompting Realities: Exploring the Potentials of Prompting for Tangible Artifacts
Mehrvarz, M., Murray-Rust, D., & Verma, H. (2025) · Biannual Conference of the Italian SIGCHI Chapter (CHItaly) · doi:10.1145/3750069.3750089
Xiong, Y., Zhang, R., Zu, Y., Liu, W., Zhu, Z., Geng, J., Li, B., Cao, Q., & Zhou, X. (2025) · IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems · doi:10.1109/tcad.2025.3624135
- Authors
- Beurer-Kellner, L., Fischer, M., & Vechev, M.
- Affiliations
- ETH Zurich (CH)
- Venue
- Proceedings of the ACM on Programming Languages
- Published
- 2023-06-06
- Volume
- 7
- Issue
- PLDI
- Pages
- 1946-1969
- ISSN
- 2475-1421 (ISSN-L)
- OpenAlex
- W4367185264
- Cited by
- 117
- References
- 13
- Citations by year
- 2026: 36 · 2025: 45 · 2024: 29 · 2023: 7
- FWCI
- 11.03
- Type
- journal article
- Language
- en
- Open access
- diamond
- doi.org
- Licence
- Locations
- Proceedings of the ACM on Programming Languages (publishedVersion, cc-by)
- arXiv (Cornell University) (submittedVersion)
- Repository for Publications and Research Data (ETH Zurich) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- ETH Zürich Research Collection
- Funders
- Swiss State Secretariat for Education, Research and Innovation (101045816)
- Topics
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Physical Sciences › Computer Science › Artificial Intelligence › Natural Language Processing Techniques
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Keywords
- prompting, large language models, output constraints, inference efficiency, cost reduction
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
