The DOI matches its Crossref record; 34 of the 34 references it cites were found in a registry or an index.
The Programmer’s Assistant: Conversational Interaction with a Large Language Model for Software Development
Open accessAbstract
Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model’s responses. We developed a prototype system – the Programmer’s Assistant – in order to explore the utility of conversational interactions grounded in code, as well as software engineers’ receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant’s capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2022) · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · doi:10.1109/cvpr52688.2022.01042 · Cited by 20,302
Rajkomar, A., Dean, J., & Kohane, I. (2019) · New England Journal of Medicine · doi:10.1056/nejmra1814259 · Cited by 4,579
ELIZA—a computer program for the study of natural language communication between man and machine
Weizenbaum, J. (1966) · Communications of the ACM · doi:10.1145/365153.365168 · Cited by 4,452
Training language models to follow instructions with human feedback
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.02155 · Cited by 4,208
Machines and Mindlessness: Social Responses to Computers
Nass, C., & Moon, Y. (2000) · Journal of Social Issues · doi:10.1111/0022-4537.00153 · Cited by 4,022 · cited 1×
“” The literature on Computers as Social Agents (CASA) helps us interpret this result as it demonstrates how computers are often treated like people [56, 67].”
BackgroundOn the Opportunities and Risks of Foundation Models
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., et al. (2021) · arXiv · doi:10.48550/arxiv.2108.07258 · Cited by 2,676
Hierarchical Text-Conditional Image Generation with CLIP Latents
Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., & Chen, M. (2022) · arXiv · doi:10.48550/arxiv.2204.06125 · Cited by 2,511 · cited 1×
“Many popular text-to-image models, such as DALL-E 2 [66] and Stable Diffusion [72], operate in a one-shot fashion, in which the user specifies a prompt, clicks a button, and gets results.”
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Chatbots: History, technology, and applications
Adamopoulou, E., & Moussiades, L. (2020) · Machine Learning with Applications · doi:10.1016/j.mlwa.2020.100006 · Cited by 1,090 · cited 1×
“Using natural language to interact with technology has had a long research history [2], starting in the 1960s with pattern-matching approaches like Eliza [104], and continuing to today with state-of-the-art large language modelbased conversational systems [107] such as Meena [3] and BlenderBot [84].”
BackgroundCompetition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043
Competition-Level Code Generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Lago, A. D., et al. (2022) · arXiv · doi:10.48550/arxiv.2203.07814 · Cited by 1,043
Machines as teammates: A research agenda on AI in team collaboration
Seeber, I., Bittner, E., Briggs, R. O., de Vreede, T., de Vreede, G.-J., Elkins, A., Maier, R., Merz, A. B., Oeste-Reiß, S., Randrup, N., et al. (2020) · Information & Management · doi:10.1016/j.im.2019.103174 · Cited by 835
LaMDA: Language Models for Dialog Applications
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. (2022) · arXiv · doi:10.48550/arxiv.2201.08239 · Cited by 778
High-Resolution Image Synthesis with Latent Diffusion Models
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., & Ommer, B. (2021) · arXiv · doi:10.48550/arxiv.2112.10752 · Cited by 737
Vaithilingam, P., Zhang, T., & Glassman, E. L. (2022) · CHI Conference on Human Factors in Computing Systems Extended Abstracts · doi:10.1145/3491101.3519665 · Cited by 660
Towards a Human-like Open-Domain Chatbot
Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., et al. (2020) · arXiv · doi:10.48550/arxiv.2001.09977 · Cited by 614
Achieving Saturation in Thematic Analysis: Development and Refinement of a Codebook
Ando, H., Cousins, R., & Young, C. (2014) · Comprehensive Psychology · doi:10.2466/03.cp.3.4 · Cited by 577
Chaves, A. P., & Gerosa, M. A. (2021) · International Journal of Human–Computer Interaction · doi:10.1080/10447318.2020.1841438 · Cited by 572
Automation bias and verification complexity: a systematic review
Lyell, D., & Coiera, E. (2017) · Journal of the American Medical Informatics Association · doi:10.1093/jamia/ocw105 · Cited by 451
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 27×
“Developers would communicate with Copilot by typing comments and seeing what Copilot generated in response.”
BackgroundInfluentialHuman-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342
Self-citationExample-centric programming: integrating web search into the development environment
Brandt, J., Dontcheva, M., Weskamp, M., & Klemmer, S. R. (2010) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/1753326.1753402 · Cited by 324
An empirical evaluation of GitHub copilot's code suggestions
Nguyen, N., & Nadi, S. (2022) · Proceedings of the 19th International Conference on Mining Software Repositories · doi:10.1145/3524842.3528470 · Cited by 313 · cited 1×
“this model have shown that, although the quality of its outputs is quite good, those outputs may still be problematic [57].”
BackgroundProductivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Li, X. A., Rice, A., Rifkin, D., Simister, S., Sittampalam, G., & Aftandilian, E. (2022) · Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming · doi:10.1145/3520312.3534864 · Cited by 200
Productivity Assessment of Neural Code Completion
Ziegler, A., Kalliamvakou, E., Simister, S., Sittampalam, G., Li, A., Rice, A., Rifkin, D., & Aftandilian, E. (2022) · arXiv · doi:10.48550/arxiv.2205.06537 · Cited by 200
Nonvisual presentation of graphical user interfaces: contrasting two approaches
Mynatt, E. D., & Weber, G. (1994) · Proceedings of the SIGCHI Conference on Human Factors in Computing Systems · doi:10.1145/191666.191732 · Cited by 156
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 2×
“[103, 105, 109]) conducted quantitative examinations of the use of LLMs in code work; our study is akin to Weisz et al.”
BackgroundMethodWang, Q., Saha, K., Gregori, E., Joyner, D., & Goel, A. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445645 · Cited by 120
Conversational UX Design: A Practitioner's Guide to the Natural Conversation Framework
Research-Almaden, I., Moore, R. J., Arar, R., & Research-Almaden, I. (2019) · doi:10.1145/3304087 · Cited by 118 · cited 4×
“We then describe our analysis of the conversation logs and our development of a coding guide based on Conversation Analysis [76], and specifically, Moore and Arar’s Natural Conversation Framework [50].”
BackgroundMethodInfluentialActionable UI Design Guidelines for Smartphone Applications Inclusive of Low-Literate Users
Srivastava, A., Kapania, S., Tuli, A., & Singh, P. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449210 · Cited by 58
AI-Assisted Human Labeling: Batching for Efficiency without Overreliance
Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., et al. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449163 · Cited by 45
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 27×
“Developers would communicate with Copilot by typing comments and seeing what Copilot generated in response.”
BackgroundInfluentialA Test for Evaluating Performance in Human-Computer Systems
Campero, A., Vaccaro, M., Song, J., Wen, H., Almaatouq, A., & Malone, T. W. (2022) · arXiv · doi:10.48550/arxiv.2206.12390 · Cited by 21
Research works printed but not linked: 68
- 10.1109/MS.2017.31 · DOI
- Safinah Ali Nisha Elizabeth Devasia and Cynthia Breazeal. 2022. Escape! Bot: Social Robots as Creative Problem-Solving Partners. In Creativity and Cognition. 275–283. · DOI
- 10.1145/3212695 · DOI · Paper page
- 10.1109/VISSOFT.2013.6650527 · DOI
- 10.1145/257089.257151 · DOI
- Amanda Askell Yuntao Bai Anna Chen Dawn Drain Deep Ganguli Tom Henighan Andy Jones Nicholas Joseph Ben Mann Nova DasSarma 2021. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861(2021). · arXiv
- 10.1145/3209978.3210027 · DOI
- Sallyann Bryant, Pablo Romero, and Benedict" du Boulay. 2006. The Collaborative Nature of Pair Programming. In Extreme Programming and Agile Processes in Software Engineering, Pekka Abrahamsson, Michele Marchesi, and Giancarlo Succi (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 53–64.
- Gaetano Cascini Yukari Nagai Georgi V Georgiev Jader Zelaya Niccolò Becattini Jean-François Boujut Hernan Casakin Nathan Crilly Elies Dekoninck John Gero 2022. Perspectives on design creativity and innovation research: 10 years later. 30 pages. · DOI
- Stephen Cass. 2022. Top Programming Languages 2022. IEEE Spectrum (23 Aug 2022). https://spectrum.ieee.org/top-programming-languages-2022
- Cristina Catalan Aguirre Nuria Gonzalez Castro Carlos Delgado Kloos Carlos Alario-Hoyos and Pedro José Muñoz Merino. 2021. Conversational agent for supporting learners on a MOOC on programming with Java. (2021). · DOI
- 10.1145/966789.966803 · DOI
- 10.1109/APSEC.2005.22 · DOI
- 10.1007/978-3-642-39646-5_8 · DOI
- 10.1145/1066129.1066132 · DOI
- Bobbie Eicher, Kathryn Cunningham, Sydni Peterson Marissa Gonzales, and Ashok Goel. 2017. Toward mutual theory of mind as a foundation for co-creation. In International Conference on Computational Creativity, Co-Creation Workshop.
- Stephen M Fiore, Eduardo Salas, and Janis A Cannon-Bowers. 2001. Group dynamics and shared mental model development. How people evaluate others in organizations 234 (2001).
- Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. 1995. Design patterns: elements of reusable object-oriented software. Addison-Wesley.
- Amelia Glaese Nat McAleese Maja Trębacz John Aslanides Vlad Firoiu Timo Ewalds Maribeth Rauh Laura Weidinger Martin Chadwick Phoebe Thacker Lucy Campbell-Gillingham Jonathan Uesato Po-Sen Huang Ramona Comanescu Fan Yang Abigail See Sumanth Dathathri Rory Greig Charlie Chen Doug Fritz Jaume Sanchez Elias Richard Green Soňa Mokrá Nicholas Fernando Boxi Wu Rachel Foley Susannah Young Iason Gabriel William Isaac John Mellor Demis Hassabis Koray Kavukcuoglu Lisa Anne Hendricks and Geoffrey Irving. 2022. Improving alignment of dialogue agents via targeted human judgements. https://arxiv.org/abs/2209.14375 · arXiv
- 10.14236/ewic/HCI2018.103 · DOI
- 10.1007/s10664-019-09730-9 · DOI · Paper page
- Edwin L Hutchins, James D Hollan, and Donald A Norman. 1985. Direct manipulation interfaces. Human–computer interaction 1, 4 (1985), 311–338.
- 10.18653/v1/P16-1195 · DOI · Paper page
- Andreas Jedlitschka and Markus Nick. 2003. Software Engineering Knowledge Repositories. Springer Berlin Heidelberg, Berlin, Heidelberg, 55–80.
- 10.1145/3411764.3445659 · DOI · Paper page
- Lauramaria Laine. 2021. Exploring Advertising Creatives’ Attitudes Towards Human-AI Collaboration. (2021).
- 10.1002/hbe2.266 · DOI
- 10.1145/263552.263612 · DOI
- John E Mathieu, Tonia S Heffner, Gerald F Goodwin, Eduardo Salas, and Janis A Cannon-Bowers. 2000. The influence of shared mental models on team process and performance.Journal of applied psychology 85, 2 (2000), 273.
- Ekaterina A Moroz, Vladimir O Grizkevich, and Igor M Novozhilov. 2022. The Potential of Artificial Intelligence as a Method of Software Developer’s Productivity Improvement. In 2022 Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus). IEEE, 386–390.
- Michael Muller, Stevean Ross, Stephanie Houde, Mayank Agarwal, Fernando Martinez, John Richards, Kartik Talamadupula, and Justin D Weisz. 2022. Drinking Chai with Your (AI) Programming Partner: A Design Fiction about Generative AI for Software Engineering. HAI-GEN Workshop at IUI 2022: 3rd Workshop on Human-AI Co-Creation with Generative Models (2022). https://hai-gen.github.io/2022/
- 10.1145/2662253.2662300 · doi:10.1145/2662253.2662300
- 10.1145/3126594.3126628 · DOI
- Martin Nordio H Estler Carlo A Furia Bertrand Meyer 2011. Collaborative software development on the web. arXiv preprint arXiv:1105.0768(2011). · arXiv
- Maxwell Nye, Anders Andreassen, Guy Gur-Ari, Henryk Witold Michalewski, Jacob Austin, David Bieber, David Martin Dohan, Aitor Lewkowycz, Maarten Paul Bosma, David Luan, Charles Sutton, and Augustus Odena. 2021. Show Your Work: Scratchpads for Intermediate Computation with Language Models. https://arxiv.org/abs/2112.00114. · arXiv
- Peter Pirolli and Stuart Card. 1999. Information foraging.Psychological review 106, 4 (1999), 643.
- 10.1145/92755.92783 · doi:10.1145/92755.92783
- Mawarny Md Rejab, James Noble, and George Allan. 2014. Distributing Expertise in Agile Software Development Projects. In 2014 Agile Conference. 33–36.
- Jeba Rezwana and Mary Lou Maher. 2021. COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems.. In ICCC. 444–448.
- Charles H. Rich and Richard C. Waters. 1990. The Programmer’s Apprentice. Addison-Wesley Publishing Company, Reading, MA.
- 10.1145/3498326 · DOI
- 10.1145/964442.964518 · doi:10.1145/964442.964518
- 10.1145/964442.964536 · DOI
- Baptiste Roziere, Marie-Anne Lachaux, Lowik Chanussot, and Guillaume Lample. 2020. Unsupervised Translation of Programming Languages. In Advances in Neural Information Processing Systems, H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.). Vol. 33. Curran Associates, Inc., 20601–20611.
- Harvey Sacks. 1984. Notes on methodology. In Structures of Social Action: Studies in Conversation Analysis, John Heritageand J. Maxwell Atkinson (Eds.). Cambridge University Press, Cambridge, 2–27.
- Nithya Sambasivan and Rajesh Veeraraghavan. 2022. The Deskilling of Domain Expertise in AI Development. In CHI Conference on Human Factors in Computing Systems. 1–14.
- 10.1145/3411764.3445544 · DOI
- 10.1177/1555343416682891 · DOI
- 10.1145/1180875.1180890 · DOI
- 10.17705/1thci.00131 · DOI · Paper page
- Ben Shneiderman. 2022. Human-Centered AI. Oxford University Press. · DOI · Paper page
- Kurt Shuster Jing Xu Mojtaba Komeili Da Ju Eric Michael Smith Stephen Roller Megan Ung Moya Chen Kushal Arora Joshua Lane 2022. BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188(2022). · arXiv
- 10.1145/3148330.3149407 · DOI
- 10.1109/MS.2012.61 · doi:10.1109/ms.2012.61
- 10.1145/2950290.2983989 · DOI
- 10.1145/3465074.3465080 · DOI
- Michele Tufano Dawn Drain Alexey Svyatkovskiy Shao Kun Deng and Neel Sundaresan. 2020. Unit Test Case Generation with Transformers and Focal Context. arXiv preprint arXiv:2009.05617(2020). · arXiv
- Severi Uusitalo, Anna Kantosalo, Antti Salovaara, Tapio Takala, and Christian Guckelsberger. 2022. Co-creative Product Design with Interactive Evolutionary Algorithms: A Practice-Based Reflection. In International Conference on Computational Intelligence in Music, Sound, Art and Design (Part of EvoStar). Springer, 292–307.
- 10.1145/3332165.3347944 · DOI
- Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Vol. 30. Curran Associates, Inc.https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
- 10.1145/3238147.3238206 · DOI · Paper page
- 10.1145/3489465 · DOI
- 10.1145/3025453.3025876 · DOI
- Justin D Weisz, Michael Muller, Stephanie Houde, John Richards, Steven I Ross, Fernando Martinez, Mayank Agarwal, and Kartik Talamadupula. 2021. Perfection Not Required? Human-AI Partnerships in Code Translation. In 26th International Conference on Intelligent User Interfaces. 402–412.
- 10.1145/3490099.3511157 · DOI · Paper page
- Aditya Ankur Yadav, Ishan Garg, and Dr. Pratistha Mathur. 2019. PACT - Programming Assistant ChaTbot. In 2019 2nd International Conference on Intelligent Communication and Computational Techniques (ICCT). 131–136.
- 10.1145/3373017.3373028 · DOI
- 10.2196/11013 · DOI
Cited by
Large language models for software engineering: survey and open problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE) · doi:10.1109/icse-fose59343.2023.00008 · Cited by 650
Large Language Models for Software Engineering: Survey and Open Problems
Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., & Zhang, J. M. (2023) · arXiv · doi:10.48550/arxiv.2310.03533 · Cited by 523
Using an LLM to Help With Code Understanding
Nam, D., Macvean, A., Hellendoorn, V., Vasilescu, B., & Myers, B. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639187 · Cited by 487
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
A Survey on Large Language Models for Code Generation
Jiang, J., Wang, F., Shen, J., Kim, S., & Kim, S. (2026) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3747588 · Cited by 251
Metadata
- Authors
- Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D.
- Venue
- Proceedings of the 28th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2023-03-27
- Pages
- 491-514
- OpenAlex
- W4321013654
- Cited by
- 258
- References
- 34
- FWCI
- 68.76
- Type
- conference paper
- Language
- en
- Open access
- hybrid
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.
