CoChat
Sign Up
Cited by 119Spreadsheets and End-User ComputingSoftware Engineering ResearchTeaching and Learning Programming

“What It Wants Me To Say”: Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models

Open access
Liu, Michael Xieyang, Sarkar, Advait, Negreanu, Carina, Zorn, Benjamin, Williams, Jack, Toronto, Neil, Gordon, Andrew D.·2023·Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
DOI 10.1145/3544548.3580817
Open this paper in CoChat — free accountOpen PDF

Abstract

Code-generating large language models map natural language to code. However, only a small portion of the infinite space of naturalistic utterances is effective at guiding code generation. For non-expert end-user programmers, learning this is the challenge of abstraction matching. We examine this challenge in the specific context of data analysis in spreadsheets, in a system that maps the user’s natural language query to Python code using the Codex generator, executes the code, and shows the result. We propose grounded abstraction matching, which bridges the abstraction gap by translating the code back into a systematic and predictable naturalistic utterance. In a between-subjects, think-aloud study (n=24), we compare grounded abstraction matching to an ungrounded alternative based on previously established query framing principles. We find that the grounded approach improves end-users’ understanding of the scope and capabilities of the code-generating model, and the kind of language needed to use it effectively.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1145/3544548.3580817

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • Designing truly human-centered GenAI systems means grappling with these metacognitive demands.

    “Designing truly human-centered GenAI systems [34,104] means grappling with these metacognitive demands.”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [104]

    Section 8.6 (Implications for design; page unavailable) recommends helping users develop mental models of how to command the system. Section 6.4 describes the decisions users make after model failures. These findings support attending to users’ cognitive demands in the code-generating interface studied, but do not establish the claim for GenAI systems generally. (LLM, conf=0.82)

    Experimental — accuracy not yet measured

  • Explainability for GenAI systems should help users adjust their confidence in their ability to prompt, evaluate outputs, and determine their automation strategy, particularly given the multiple, non-intuitive failure modes common to GenAI systems.

    “Explainability for GenAI systems should help users adjust their confidence in their ability to prompt, evaluate outputs, and determine their automation strategy, particularly given the multiple, non-intuitive failure modes common to GenAI systems, and other challenges [104].”

    From The Metacognitive Demands and Opportunities of Generative AI (2024) · cites this paper as [104]

    Split into 2 atoms — 1 confirmed, 1 partially supported, 0 disputed, 0 misattributed, 0 unverifiable. Expand the claim to see per-atom verdicts.

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

The citation lists haven’t been fetched yet

References 83 · Cited by 119

Research works printed but not linked: 110

  1. Shengnan An, Yifei Li, Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian-Guang Lou. 2022. Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Models. ArXiv abs/2203.03131 (2022). · arXiv
  2. Deniz Arsan, Ali Zaidi, Aravind Sagar, and Ranjitha Kumar. 2021. App-Based Task Shortcuts for Virtual Assistants. In The 34th Annual ACM Symposium on User Interface Software and Technology. 1089-1099.
  3. Mikhail Bakhtin and Ghodrat Ghāsemipour. 2011. The problem of speech genres. Literary Criticism 4, 15 (2011), 114-136.
  4. Emily M. Bender and Alexander Koller. 2020. Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Online, 5185-5198. https://doi.org/10.18653/v1/ 2020.acl-main.463 · DOI · Paper page
  5. Derek Bickerton. 1983. Creole languages. Scientific American 249, 1 (1983), 116-123.
  6. Derek Bickerton. 1984. The language bioprogram hypothesis. Behavioral and brain sciences 7, 2 (1984), 173-188.
  7. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77-101.
  8. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. ArXiv abs/2005.14165 (2020). · arXiv
  9. John L Campbell, Charles Quincy, Jordan Osserman, and Ove K Pedersen. 2013. Coding in-depth semistructured interviews: Problems of unitization and inter- coder reliability and agreement. Sociological methods & research 42, 3 (2013), 294-320.
  10. Federico Cassano, John Gouwar, Daniel Nguyen, Sy Duy Nguyen, Luna Phipps- Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q. Feldman, Arjun Guha, Michael Greenberg, and Abhinav Jangda. 2022. A Scalable and Extensible Approach to Benchmarking NL2Code for 18 Program- ming Languages. ArXiv abs/2208.08227 (2022). · arXiv
  11. George Chalhoub and Advait Sarkar. 2022. "It's Freedom to Put Things Where My Mind Wants": Understanding and Improving the User Experience of Struc- turing Data in Spreadsheets. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-24. https://doi.org/10.1145/3491102.3501833 · DOI
  12. Kathy Charmaz. 2006. Constructing Grounded Theory: A Practical Guide through Qualitative Analysis. SAGE. Google-Books-ID: 2ThdBAAAQBAJ.
  13. William R Cook. 2007. Applescript. In Proceedings of the third ACM SIGPLAN conference on History of programming languages. 1-1.
  14. Mihaly Csikszentmihalyi and Reed Larson. 2014. Validity and reliability of the experience-sampling method. In Flow and the foundations of positive psychology. Springer, 35-54.
  15. Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu, and Dongmei Zhang. 2019. Ta- bleSense: Spreadsheet Table Detection with Convolutional Neural Networks. Proceedings of the AAAI Conference on Artificial Intelligence 33, 01 (July 2019), 69-76. https://doi.org/10.1609/aaai.v33i01.330169 Number: 01. · DOI
  16. Paul Dourish. 2006. Implications for design. In Proceedings of the SIGCHI confer- ence on Human Factors in computing systems. 541-550.
  17. Ahmed Elgohary, Christopher Meek, Matthew Richardson, Adam Fourney, Gonzalo A. Ramos, and Ahmed Hassan Awadallah. 2021. NL-EDIT: Correcting Semantic Parse Errors through Natural Language Interaction. In NAACL.
  18. Alex Endert, Chao Han, Dipayan Maiti, Leanna House, and Chris North. 2011. Observation-level interaction with statistical models for visual analytics. In 2011 IEEE conference on visual analytics science and technology (VAST). IEEE, 121-130.
  19. Ilker Etikan, Sulaiman Abubakar Musa, Rukayya Sunusi Alkassim, et al. 2016. Comparison of convenience sampling and purposive sampling. American journal of theoretical and applied statistics 5, 1 (2016), 1-4.
  20. Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. 2022. Out of the BLEU: how should we assess quality of the Code Generation models? ArXiv abs/2208.03133 (2022). · arXiv
  21. Matthew Finlayson, Kyle Richardson, Ashish Sabharwal, and Peter Clark. 2022. What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment. ArXiv abs/2204.09148 (2022). · arXiv
  22. Marsha E Fonteyn, Benjamin Kuipers, and Susan J Grobe. 1993. A description of think aloud method and protocol analysis. Qualitative health research 3, 4 (1993), 430-441.
  23. TRG Green, M Petre, and RKE Bellamy. 1991. Comprehensibility of Vi- sual and Textual Programs: A Test of Superlativism Against the'Match- Mismatch'Conjecture. Empirical Studies of Programming: Fourth Workshop. J. Koenemann-Belliveau, TG Moher and SP Robertson. New Brunswick.
  24. Thomas R. G. Green and Marian Petre. 1996. Usability analysis of visual pro- gramming environments: a 'cognitive dimensions' framework. Journal of Visual Languages & Computing 7, 2 (1996), 131-174.
  25. Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices 46, 1 (2011), 317-330.
  26. Philip J Guo. 2013. Online python tutor: embeddable web-based program visualization for cs education. In Proceeding of the 44th ACM technical symposium on Computer science education. 579-584.
  27. Mark Halpern. 1966. Foundations of the Case for Natural-Language Program- ming. In Proceedings of the November 7-10, 1966, Fall Joint Computer Conference (San Francisco, California) (AFIPS '66 (Fall)). Association for Computing Ma- chinery, New York, NY, USA, 639-649. https://doi.org/10.1145/1464291.1464360 · DOI
  28. Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139-183.
  29. Andrew Head, Codanda Appachu, Marti A Hearst, and Björn Hartmann. 2015. Tutorons: Generating context-relevant, on-demand explanations and demon- strations of online code. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 3-12.
  30. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Xiaodong Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. ArXiv abs/2105.09938 (2021). · arXiv
  31. Andreas Holzinger. 2018. From machine learning to explainable AI. In 2018 world symposium on digital intelligence for systems and machines (DISA). IEEE, 55-66.
  32. Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2022. Instruction Induction: From Few Examples to Natural Language Task Descriptions. ArXiv abs/2205.10782 (2022). · arXiv
  33. Amber Horvath, Michael Xieyang Liu, River Hendriksen, Connor Shannon, Emma Paterson, Kazi Jawad, Andrew Macvean, and Brad A Myers. 2022. Un- derstanding How Programmers Can Use Annotations on Documentation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-16. https://doi.org/10.1145/3491102.3502095 · DOI · Paper page
  34. Jane Hsieh, Michael Xieyang Liu, Brad A. Myers, and Aniket Kittur. 2018. An Exploratory Study of Web Foraging to Understand and Support Programming Decisions. In 2018 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 305-306. https://doi.org/10.1109/VLHCC.2018.8506517 ISSN: 1943-6092. · DOI
  35. Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manip- ulation interfaces. Human-computer interaction 1, 4 (1985), 311-338.
  36. Naman Jain, Skanda Vaidyanath, Arun Iyer, Nagarajan Natarajan, Suresh Parthasarathy, Sriram Rajamani, and Rahul Sharma. 2022. Jigsaw: Large Lan- guage Models meet Program Synthesis. In International Conference on Software Engineering (ICSE). https://www.microsoft.com/en-us/research/publication/ jigsaw-large-language-models-meet-program-synthesis/
  37. Dhanya Jayagopal, Justin Lubin, and Sarah E Chasins. 2022. Exploring the Learnability of Program Synthesizers by Novice Programmers. In User Interface Software and Technology (UIST'22).
  38. Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In CHI Conference on Human Factors in Computing Systems. 1-19.
  39. Nima Joharizadeh, Advait Sarkar, Andrew D. Gordon, and Jack Williams. 2020. Gridlets: Reusing Spreadsheet Grids. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1-7. https://doi.org/10.1145/ 3334480.3382806 · DOI
  40. Daniel Khashabi, Shan Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sameer Singh, Sean Welleck, Hannaneh Hajishirzi, Tushar Khot, Ashish Sabharwal, and Yejin Choi. 2022. Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts. In NAACL.
  41. Tae Soo Kim, DaEun Choi, Yoonseo Choi, and Juho Kim. 2022. Stylette: Styling the Web with Natural Language. In CHI Conference on Human Factors in Com- puting Systems. 1-17.
  42. Amy J. Ko, Brad A. Myers, and Htet Htet Aung. 2004. Six Learning Barriers in End-User Programming Systems. In Proceedings of the 2004 IEEE Symposium on Visual Languages -Human Centric Computing (VLHCC '04). IEEE Computer Society, Washington, DC, USA, 199-206. https://doi.org/10.1109/VLHCC.2004. 47 · DOI
  43. Rafal Kocielnik, Saleema Amershi, and Paul N Bennett. 2019. Will you accept an imperfect ai? exploring designs for adjusting end-user expectations of ai systems. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1-14.
  44. Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf. 2015. Principles of explanatory debugging to personalize interactive machine learning. In Proceedings of the 20th international conference on intelligent user interfaces. 126-137.
  45. Kirby Kuznia, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Less is More: Summary of Long Instructions is Better for Program Synthesis. ArXiv abs/2203.08597 (2022). · arXiv
  46. Shuvendu K. Lahiri, Aaditya Naik, Georgios Sakkas, Piali Choudhury, Curtis von Veh, Madan Musuvathi, Jeevana Priya Inala, Chenglong Wang, and Jianfeng Gao. 2022. Interactive Code Generation via Test-Driven User-Intent Formalization. ArXiv abs/2208.05950 (2022). · arXiv
  47. Tessa Lau. 2009. Why programming-by-demonstration systems fail: Lessons learned for usable ai. AI Magazine 30, 4 (2009), 65-65.
  48. Vladimir I Levenshtein et al. 1966. Binary codes capable of correcting deletions, insertions, and reversals. In Soviet physics doklady, Vol. 10. Soviet Union, 707- 710.
  49. James R. Lewis. 2018. The System Usability Scale: Past, Present, and Future. International Journal of Human-Computer Interaction 34, 7 (July 2018), 577- 590. https://doi.org/10.1080/10447318.2018.1455307 Publisher: Taylor & Francis _eprint: https://doi.org/10.1080/10447318.2018.1455307. · DOI · Paper page
  50. Toby Jia-Jun Li, Amos Azaria, and Brad A. Myers. 2017. SUGILITE: Creat- ing Multimodal Smartphone Automation by Demonstration. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). Association for Computing Machinery, Denver, Colorado, USA, 6038-6049. https://doi.org/10.1145/3025453.3025483 · DOI
  51. Toby Jia-Jun Li, Igor Labutov, Xiaohan Nancy Li, Xiaoyi Zhang, Wenze Shi, Wanling Ding, Tom M. Mitchell, and Brad A. Myers. 2018. APPINITE: A Multi-Modal Interface for Specifying Data Descriptions in Programming by Demonstration Using Natural Language Instructions. In 2018 IEEE Sympo- sium on Visual Languages and Human-Centric Computing (VL/HCC). 105-114. https://doi.org/10.1109/VLHCC.2018.8506506 ISSN: 1943-6106. · DOI
  52. Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M. Mitchell, and Brad A. Myers. 2019. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). Association for Computing Machinery, New Orleans, LA, USA, 577-589. https://doi.org/10.1145/3332165.3347899 · DOI
  53. Yuntao Li, Bei Chen, Qian Liu, Yan Gao, Jian-Guang Lou, Yan Zhang, and Dongmei Zhang. 2020. "What Do You Mean by That?" -a Parser-Independent Interactive Approach for Enhancing Text-to-SQL. In EMNLP.
  54. Zongjie Li, Chaozheng Wang, Zhibo Liu, Haoxuan Wang, Shuai Wang, and Cuiyun Gao. 2022. CCTEST: Testing and Repairing Code Completion Systems. ArXiv abs/2208.08289 (2022). · arXiv
  55. Hugo Liu and Henry Lieberman. 2005. Programmatic Semantics for Natural Lan- guage Interfaces. In CHI '05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA) (CHI EA '05). Association for Computing Machinery, New York, NY, USA, 1597-1600. https://doi.org/10.1145/1056808.1056975 · DOI
  56. Michael Xieyang Liu, Jane Hsieh, Nathan Hahn, Angelina Zhou, Emily Deng, Shaun Burley, Cynthia Taylor, Aniket Kittur, and Brad A. Myers. 2019. Unakite: Scaffolding Developers' Decision-Making Using the Web. In Proceedings of the 32Nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). ACM, New Orleans, LA, USA, 67-80. https://doi.org/10.1145/3332165. 3347908 event-place: New Orleans, LA, USA. · DOI
  57. Michael Xieyang Liu, Aniket Kittur, and Brad A. Myers. 2022. Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3491102.3501968 event-place: New Orleans, LA, USA. · DOI
  58. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. ArXiv abs/2107.13586 (2021). · arXiv
  59. Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA" The Gulf between User Expectation and Experience of Conversational Agents. In Pro- ceedings of the 2016 CHI conference on human factors in computing systems. 5286-5297.
  60. Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Zi- heng Huang. 2022. Generating Diverse Code Explanations using the GPT-3 Large Language Model. In Proceedings of the 2022 ACM Conference on International Computing Education Research -Volume 2 (ICER '22). Association for Computing Machinery, New York, NY, USA, 37-39. https://doi.org/10.1145/3501709.3544280 · DOI · Paper page
  61. Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability and inter-rater reliability in qualitative research: Norms and guidelines for CSCW and HCI practice. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1-23.
  62. Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. Reframing Instructional Prompts to GPTk's Language. arXiv preprint arXiv:2109.07830 (2021). · arXiv
  63. Lingbo Mo, Ashley Glen Lewis, Huan Sun, and Michael White. 2022. Towards Transparent Interactive Semantic Parsing via Step-by-Step Correction. ArXiv abs/2110.08345 (2022). · arXiv
  64. Jesse Mu and Advait Sarkar. 2019. Do we need natural language? Exploring restricted language interfaces for complex domains. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1-6.
  65. Avanika Narayan, Ines Chami, Laurel Orr, and Christopher R'e. 2022. Can Foundation Models Wrangle Your Data? ArXiv abs/2205.09911 (2022). · arXiv
  66. Avanika Narayan, Ines Chami, Laurel Orr, and Christopher R'e. 2022. Can Foundation Models Wrangle Your Data? ArXiv abs/2205.09911 (2022). · arXiv
  67. Petr Necesal and Jan Pospıšil. 2012. Experience with teaching mathematics for engineers with the aid of Wolfram Alpha. In Proceedings of the World Congress on Engineering and Computer Science, Vol. 1. 271-274.
  68. Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Haiquan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. A Conversational Paradigm for Program Synthesis. ArXiv abs/2203.13474 (2022). · arXiv
  69. Fatma Őzcan, Abdul Quamar, Jaydeep Sen, Chuan Lei, and Vasilis Efthymiou. 2020. State of the art and open challenges in natural language interfaces to data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. 2629-2636.
  70. Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, M. Hassan Murad, and Chitta Baral. 2022. In-BoXBART: Get Instructions into Biomedical Multi-Task Learning. In NAACL-HLT.
  71. Pruthvi Patel, Swaroop Mishra, Mihir Parmar, and Chitta Baral. 2022. Is a Question Decomposition Unit All We Need? ArXiv abs/2205.12538 (2022). · arXiv
  72. Mitchel Resnick, John Maloney, Andrés Monroy-Hernández, Natalie Rusk, Eve- lyn Eastmond, Karen Brennan, Amon Millner, Eric Rosenbaum, Jay Silver, Brian Silverman, et al. 2009. Scratch: programming for all. Commun. ACM 52, 11 (2009), 60-67.
  73. John Rieman. 1993. The diary study: a workplace-oriented research tool to guide laboratory efforts. In Proceedings of the INTERACT'93 and CHI'93 conference on Human factors in computing systems. 321-326.
  74. Jean E. Sammet. 1966. The Use of English as a Programming Language. Commun. ACM 9, 3 (mar 1966), 228-230. https://doi.org/10.1145/365230.365274 · DOI
  75. Advait Sarkar. 2018. Interactive analytical modelling. Technical Report. Univer- sity of Cambridge, Computer Laboratory.
  76. Advait Sarkar. 2022. Is explainable AI a race against model complexity?. In Work- shop on Transparency and Explanations in Smart Systems (TeXSS), in conjunction with ACM Intelligent User Interfaces (IUI 2022) (CEUR Workshop Proceedings, 3124). 192-199. http://ceur-ws.org/Vol-3124/paper22.pdf
  77. Advait Sarkar, Alan F Blackwell, Mateia Jamnik, and Martin Spott. 2014. Teach and try: A simple interaction technique for exploratory data modelling by end users. In 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 53-56.
  78. Advait Sarkar, Alan F. Blackwell, Mateja Jamnik, and Martin Spott. 2015. In- teraction with Uncertainty in Visualisations. (2015). https://doi.org/10.2312/ eurovisshort.20151138 Accepted: 2015-05-24T19:43:20Z Publisher: The Euro- graphics Association. · DOI
  79. Advait Sarkar, Judith W. Borghouts, Anusha Iyer, Sneha Khullar, Christian Can- ton, Felienne Hermans, Andrew D. Gordon, and Jack Williams. 2020. Spread- sheet Use and Programming Experience: An Exploratory Survey. In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems (CHI EA '20). Association for Computing Machinery, New York, NY, USA, 1-9. https://doi.org/10.1145/3334480.3382807 · DOI
  80. Advait Sarkar and Andrew D. Gordon. 2018. How do people learn to use spreadsheets? (Work in progress). In Proceedings of the 29th Annual Conference of the Psychology of Programming Interest Group (PPIG 2018). 28-35.
  81. Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srinivasa Ragavan, and Ben Zorn. 2022. What is it like to program with ar- tificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]. · DOI · Paper page
  82. Advait Sarkar, Mateja Jamnik, Alan F. Blackwell, and Martin Spott. 2015. In- teractive visual machine learning in spreadsheets. In 2015 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). 159-163. https: //doi.org/10.1109/VLHCC.2015.7357211 · DOI
  83. Advait Sarkar, Sruti Srinivasa Ragavan, Jack Williams, and Andrew D Gordon. 2022. End-user encounters with lambda abstraction in spreadsheets: Apollo's bow or Achilles' heel?. In 2022 IEEE Symposium on Visual Languages and Human- Centric Computing (VL/HCC). IEEE Computer Society, 1-11.
  84. Advait Sarkar, Martin Spott, Alan F Blackwell, and Mateja Jamnik. 2016. Visual discovery and model-driven explanation of time series patterns. In 2016 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 78-86.
  85. Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. 2016. Vega-lite: A grammar of interactive graphics. IEEE transactions on visualization and computer graphics 23, 1 (2016), 341-350.
  86. Christopher Scaffidi, Mary Shaw, and Brad Myers. 2005. Estimating the numbers of end users and end user programmers. In 2005 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC'05). IEEE, 207-214.
  87. Viktor Schlegel, Benedikt Lang, Siegfried Handschuh, and André Freitas. 2019. Vajra: step-by-step programming with natural language. In Proceedings of the 24th International Conference on Intelligent User Interfaces. 30-39.
  88. Vidya Setlur and Melanie Tory. 2022. How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behavior. In CHI Conference on Human Factors in Computing Systems. 1-17.
  89. Disha Shrivastava, H. Larochelle, and Daniel Tarlow. 2022. Repository-Level Prompt Generation for Large Language Models of Code. ArXiv abs/2206.12839 (2022). · arXiv
  90. Ankita Nandkishor Sontakke, Manasi Patwardhan, Lovekesh Vig, Raveendra Ku- mar Medicherla, Ravindra Naik, and Gautam Shroff. 2022. Code Summarization: Do Transformers Really Understand Code?. In Deep Learning for Code Workshop.
  91. Sruti Srinivasa Ragavan, Zhitao Hou, Yun Wang, Andrew D Gordon, Haidong Zhang, and Dongmei Zhang. 2022. GridBook: Natural Language Formulas for the Spreadsheet Grid. In 27th International Conference on Intelligent User Interfaces (IUI '22). Association for Computing Machinery, New York, NY, USA, 345-368. https://doi.org/10.1145/3490099.3511161 · DOI · Paper page
  92. Erik Stolterman. 2008. The nature of design practice and implications for interaction design research. International Journal of Design 2, 1 (2008).
  93. Harry R. Tennant, Kenneth M. Ross, and Craig W. Thompson. 1983. Usable Natu- ral Language Interfaces through Menu-Based Natural Language Understanding. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, Massachusetts, USA) (CHI '83). Association for Computing Machinery, New York, NY, USA, 154-160. https://doi.org/10.1145/800045.801601 · DOI
  94. Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kul- shreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kris- ten Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022. LaMDA: Language Models for Dialog Applications. (Jan. 2022). https://doi.org/10.48550/arXiv.2201.08239 · DOI · Paper page
  95. Immanuel Trummer. 2022. CodexDB: Generating Code for Processing SQL Queries using GPT-3 Codex. ArXiv abs/2204.08941 (2022). · arXiv
  96. Mojtaba Vaismoradi, Hannele Turunen, and Terese Bondas. 2013. Content anal- ysis and thematic analysis: Implications for conducting a qualitative descriptive study. Nursing & Health Sciences 15, 3 (2013), 398-405. https://doi.org/10.1111/ nhs.12048 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/nhs.12048. · DOI · Paper page
  97. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems 30 (2017).
  98. Chaozheng Wang, Yuan-Hong Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R. Lyu. 2022. No More Fine-Tuning? An Experimental Evaluation of Prompt Tuning in Code Intelligence. ArXiv abs/2207.11680 (2022). · arXiv
  99. Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Yu Wu, and Weiran Xu. 2022. InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER. ArXiv abs/2203.03903 (2022). · arXiv
  100. Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2022. MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages. ArXiv abs/2203.08388 (2022). · arXiv
  101. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682 (2022). · arXiv
  102. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of Thought Prompting Elicits Reasoning in Large Language Models. ArXiv abs/2201.11903 (2022). · arXiv
  103. Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfec- tion not required? Human-AI partnerships in code translation. In 26th Interna- tional Conference on Intelligent User Interfaces. 402-412.
  104. Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. 2022. PromptChainer: Chaining Large Lan- guage Model Prompts through Visual Programming. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems (CHI EA '22). Association for Computing Machinery, New York, NY, USA, 1-10. https://doi.org/10.1145/3491101.3519729 · DOI · Paper page
  105. Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022. AI Chains: Trans- parent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts. In CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1-22. https://doi.org/10.1145/3491102.3517582 · DOI
  106. Ziyu Yao, Yu Su, Huan Sun, and Wen tau Yih. 2019. Model-based Interactive Semantic Parsing: A Unified Framework and A Text-to-SQL Case Study. In EMNLP.
  107. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. https://doi.org/10. 48550/arXiv.1809.08887 arXiv:1809.08887 [cs]. · DOI · Paper page
  108. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir R. Radev. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 -November 4, 2018, Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun'ichi Tsujii (Eds.). Association for Computational Linguistics, 3911-3921. https://doi.org/10.18653/v1/d18-1425 · DOI · Paper page
  109. Xiong Zhang and Philip J Guo. 2017. Ds. js: Turn any webpage into an example- centric live programming environment for learning data science. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 691-702.
  110. Ming Zhu, Aneesh Jain, Karthik Suresh, Roshan Ravindran, Sindhu Tipirneni, and Chandan K. Reddy. 2022. XLCoST: A Benchmark Dataset for Cross-lingual Code Intelligence. ArXiv abs/2206.08474 (2022). · arXiv

Authors
Liu, M. X., Sarkar, A., Negreanu, C., Zorn, B., Williams, J., Toronto, N., & Gordon, A. D.
Affiliations
Microsoft (United States) (US)
ORCID
Venue
Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems
Publisher
ACM
Published
2023-04-19
Pages
1-31
DOI
10.1145/3544548.3580817
OpenAlex
W4365601419
Cited by
119
References
83
Citations by year
2026: 24 · 2025: 39 · 2024: 47 · 2023: 9
FWCI
23.02
Type
conference paper
Language
en
Open access
Licence
www.acm.org/publications/policies/copyright_policy#Background (vor)
Locations
  • CHI Conference on Human Factors in Computing Systems (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Software › Spreadsheets and End-User Computing
  • Physical Sciences › Computer Science › Information Systems › Software Engineering Research
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
Keywords
code generation, large language models, end-user programmers, natural language to code, spreadsheet analysis, Python code generation, codices, think-aloud study

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review