CoChat
Sign Up
Cited by 13Software Engineering Techniques and PracticesSoftware Engineering ResearchSoftware Testing and Debugging Techniques

Beyond the Comfort Zone: Emerging Solutions to Overcome Challenges in Integrating LLMs into Software Products

Open access
Nahar, Nadia, Kästner, Christian, Butler, Jenna, Parnin, Chris, Zimmermann., Thomas, Bird, Christian·2025·2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
DOI 10.1109/icse-seip66354.2025.00051
Open this paper in CoChat — free accountOpen PDF

Abstract

Large Language Models (LLMs) are increasingly embedded into software products across diverse industries, en-hancing user experiences, but at the same time introducing numerous challenges for developers. Unique characteristics of LLMs force developers, who are accustomed to traditional software development and evaluation, out of their comfort zones as the LLM components shatter standard assumptions about software systems. This study explores the emerging solutions that software developers are adopting to navigate the encountered challenges. Leveraging a mixed-method research, including 26 interviews and a survey with 182 responses, the study identifies 19 emerging solutions regarding quality assurance that practitioners across several product teams at Microsoft are exploring. The findings provide valuable insights that can guide the development and evaluation of LLM-based products more broadly in the face of these challenges.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1109/icse-seip66354.2025.00051

Cites

7 of 7 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Nahar et al. [42] highlighted needs regarding standardising responsible AI evaluation to mitigate ethical and legal issues for organisations.

    “Such needs were also highlighted by Nahar et al. [42] regarding standardising responsible AI evaluation to mitigate ethical and legal issues for organisations.”

    From Same Scrutiny, More Time: Eye Tracking Insights into Reviewing LLM-Labelled Code (2026) · cites this paper as [42]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Research works printed but not linked: 41

  1. J. T. Liang, M. Lin, N. Rao, and B. A. Myers, "Prompts are programs too! understanding how developers build software containing prompts," arXiv preprint arXiv:2409.12447, 2024. · arXiv
  2. C. Parnin, G. Soares, R. Pandita, S. Gulwani, J. Rich, and A. Z. Henley, "Building your own product copilot: Challenges, opportunities, and needs," arXiv preprint arXiv:2312.14231, 2023. · arXiv
  3. N. Nahar, H. Zhang, G. Lewis, S. Zhou, and C. Kästner, "A meta- summary of challenges in building products with ml components- collecting experiences from 4758+ practitioners," in 2023 IEEE/ACM 2nd International Conference on AI Engineering-Software Engineering for AI (CAIN), pp. 171-183, IEEE, 2023.
  4. A. E. Hassan, G. A. Oliva, D. Lin, B. Chen, Z. Ming, et al., "Re- thinking software engineering in the foundation model era: From task- driven ai copilots to goal-driven ai pair programmers," arXiv preprint arXiv:2404.10225, 2024. · arXiv
  5. M. Dolata, N. Lange, and G. Schwabe, "Development in times of hype: How freelancers explore generative ai?," in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pp. 1-13, 2024.
  6. T. S. Kim, Y. Lee, J. Shin, Y.-H. Kim, and J. Kim, "Evallm: Interactive evaluation of large language model prompts on user-defined criteria," in Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2024.
  7. A. Mishra, U. Soni, A. Arunkumar, J. Huang, B. C. Kwon, and C. Bryan, "Promptaid: Prompt exploration, perturbation, testing and it- eration using visual analytics for large language models," arXiv preprint arXiv:2304.01964, 2023. · arXiv
  8. E. Reif, M. Kahng, and S. Petridis, "Visualizing linguistic diversity of text datasets synthesized by large language models," in 2023 IEEE Visualization and Visual Analytics (VIS), pp. 236-240, IEEE, 2023.
  9. S. Shankar, H. Li, P. Asawa, M. Hulsebos, Y. Lin, J. Zamfirescu- Pereira, H. Chase, W. Fu-Hinthorn, A. G. Parameswaran, and E. Wu, "Spade: Synthesizing assertions for large language model pipelines," arXiv preprint arXiv:2401.03038, 2024. · arXiv
  10. B. Murugadoss, C. Poelitz, I. Drosos, V. Le, N. McKenna, C. S. Negreanu, C. Parnin, and A. Sarkar, "Evaluating the evaluator: Mea- suring llms' adherence to task evaluation instructions," arXiv preprint arXiv:2408.08781, 2024. · arXiv
  11. A. O'Cathain, "Assessing the quality of mixed methods research: Toward a comprehensive framework," Handbook of mixed methods in social and behavioral research, vol. 531, p. 555, 2010.
  12. J. W. Creswell, V. L. P. Clark, M. L. Gutmann, and W. E. Hanson, "Ad- vanced mixed method research designs," Handbook of mixed methods in social & behavioral research, vol. 209, pp. 209-240, 2003.
  13. C. Kästner, Machine Learning in Production: From Models to Products. MIT Press, in production, 2024.
  14. T. Wu, E. Jiang, A. Donsbach, J. Gray, A. Molina, M. Terry, and C. J. Cai, "Promptchainer: Chaining large language model prompts through visual programming," in CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1-10, 2022.
  15. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., "Retrieval- augmented generation for knowledge-intensive nlp tasks," Advances in Neural Information Processing Systems, vol. 33, pp. 9459-9474, 2020.
  16. O. Topsakal and T. C. Akinci, "Creating large language model appli- cations utilizing langchain: A primer on developing llm apps fast," in International Conference on Applied Engineering and Natural Sciences, vol. 1, pp. 1050-1056, 2023.
  17. C. Jeong, "Generative ai service implementation using llm application architecture: based on rag model and langchain framework," Journal of Intelligence and Information Systems, vol. 29, no. 4, pp. 129-164, 2023.
  18. P. Sahoo, A. K. Singh, S. Saha, V. Jain, S. Mondal, and A. Chadha, "A systematic survey of prompt engineering in large language models: Techniques and applications," arXiv preprint arXiv:2402.07927, 2024. · arXiv
  19. B. Chen, Z. Zhang, N. Langrené, and S. Zhu, "Unleashing the potential of prompt engineering in large language models: a comprehensive review," arXiv preprint arXiv:2310.14735, 2023. · arXiv
  20. S. Vatsal and H. Dubey, "A survey of prompt engineering meth- ods in large language models for different nlp tasks," arXiv preprint arXiv:2407.12994, 2024. · arXiv
  21. O. Khattab, A. Singhvi, P. Maheshwari, Z. Zhang, K. Santhanam, S. Haq, A. Sharma, T. T. Joshi, H. Moazam, H. Miller, et al., "Dspy: Compiling declarative language model calls into state-of-the-art pipelines," in The Twelfth International Conference on Learning Representations, 2024.
  22. T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh, "Auto- prompt: Eliciting knowledge from language models with automatically generated prompts," arXiv preprint arXiv:2010.15980, 2020. · arXiv
  23. R. Pryzant, D. Iter, J. Li, Y. T. Lee, C. Zhu, and M. Zeng, "Automatic prompt optimization with" gradient descent" and beam search," in In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222--4235, 2020.
  24. T. Dixit, D. Lee, S. Fang, S. S. Harsha, A. Sureshan, A. Maharaj, and Y. Li, "Retain: Interactive tool for regression testing guided llm migration," arXiv preprint arXiv:2409.03928, 2024. · arXiv
  25. S. Wu, H. Shen, D. S. Weld, J. Heer, and M. T. Ribeiro, "Scattershot: Interactive in-context example curation for text transformation," in Proceedings of the 28th International Conference on Intelligent User Interfaces, pp. 353-367, 2023.
  26. S. Petridis, B. D. Wedin, J. Wexler, M. Pushkarna, A. Donsbach, N. Goyal, C. J. Cai, and M. Terry, "Constitutionmaker: Interactively critiquing large language models by converting feedback into principles," in Proceedings of the 29th International Conference on Intelligent User Interfaces, pp. 853-868, 2024.
  27. K. I. Gero, J. K. Kummerfeld, and E. L. Glassman, "Sensemaking interfaces for human evaluation of language model outputs," in NeurIPS: In Workshop on Human Evaluation of Generative Models, 2022.
  28. L. Aroyo and C. Welty, "Truth is a lie: Crowd truth and the seven myths of human annotation," AI Magazine, vol. 36, no. 1, pp. 15-24, 2015.
  29. J. Zamfirescu-Pereira, R. Y. Wong, B. Hartmann, and Q. Yang, "Why johnny can't prompt: how non-ai experts try (and fail) to design llm prompts," in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-21, 2023.
  30. W. Ma, C. Yang, and C. Kästner, "(why) is my prompt getting worse? rethinking regression testing for evolving llm apis," in Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering-Software Engineering for AI, pp. 166-171, 2024.
  31. M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr, "Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting," arXiv preprint arXiv:2310.11324, 2023. · arXiv
  32. S. Campbell, M. Greenwood, S. Prior, T. Shearer, K. Walkem, S. Young, D. Bywaters, and K. Walker, "Purposive sampling: complex or simple? research case examples," Journal of research in Nursing, vol. 25, no. 8, pp. 652-661, 2020.
  33. C. Parker, S. Scott, and A. Geddes, "Snowball sampling," SAGE research methods foundations, 2019.
  34. H.-F. Hsieh and S. E. Shannon, "Three approaches to qualitative content analysis," Qualitative health research, vol. 15, no. 9, pp. 1277-1288, 2005.
  35. J. Lazar, J. H. Feng, and H. Hochheiser, Research methods in human- computer interaction. Morgan Kaufmann, 2017.
  36. T. Punter, M. Ciolkowski, B. Freimut, and I. John, "Conducting on-line surveys in software engineering," in Proceedings of the International Symposium on Empirical Software Engineering, pp. 80-88, IEEE, 2003.
  37. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al., "Judging llm-as-a-judge with mt-bench and chatbot arena," Advances in Neural Information Processing Systems, vol. 36, pp. 46595-46623, 2023.
  38. R. Kamoi, S. S. S. Das, R. Lou, J. J. Ahn, Y. Zhao, X. Lu, N. Zhang, Y. Zhang, R. H. Zhang, S. R. Vummanthala, et al., "Evaluating llms at detecting errors in llm responses," arXiv preprint arXiv:2404.03602, 2024. · arXiv
  39. K. Claessen and J. Hughes, "Quickcheck: a lightweight tool for random testing of haskell programs," in Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, pp. 268-279, 2000.
  40. A. Zeller, Why programs fail: a guide to systematic debugging. Morgan Kaufmann, 2009.
  41. B. Rakova, J. Yang, H. Cramer, and R. Chowdhury, "Where responsible ai meets reality: Practitioner perspectives on enablers for shifting or- ganizational practices," Proceedings of the ACM on Human-Computer Interaction, vol. 5, no. CSCW1, pp. 1-23, 2021.

Cited by

Sorted by citations

Authors
Nahar, N., Kästner, C., Butler, J., Parnin, C., Zimmermann., T., & Bird, C.
Affiliations
  • Carnegie Mellon University (US)
  • Microsoft (United States) (US)
  • Microsoft Research
ORCID
Venue
2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP)
Publisher
IEEE
Published
2025-04-27
Pages
516-527
DOI
10.1109/icse-seip66354.2025.00051
OpenAlex
W4413349822
Cited by
13
References
7
Citations by year
2026: 6 · 2025: 7
FWCI
11.93
Type
conference paper
Language
en
Open access
Licence
Locations
  • IEEE/ACM International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Software › Software Testing and Debugging Techniques
Keywords
large language models, developer challenges, quality assurance, LLM evaluation, developer practices

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review