Better Together? An Evaluation of AI-Supported Code Translation
Open accessThe DOI matches its Crossref record; 103 of the 104 references it cites were found in a registry or an index, and 1 are not checked yet.
Abstract
Generative machine learning models have recently been applied to source code, for use cases including translating code between programming languages, creating documentation from code, and auto-completing methods. Yet, state-of-the-art models often produce code that is erroneous or incomplete. In a controlled study with 32 software engineers, we examined whether such imperfect outputs are helpful in the context of Java-to-Python code translation. When aided by the outputs of a code translation model, participants produced code with fewer errors than when working alone. We also examined how the quality and quantity of AI translations affected the work process and quality of outcomes, and observed that providing multiple translations had a larger impact on the translation process than varying the quality of provided translations. Our results tell a complex, nuanced story about the benefits of generative code models and the challenges software engineers face when working with their outputs. Our work motivates the need for intelligent user interfaces that help software engineers effectively work with generative code models in order to understand and evaluate their outputs and achieve superior outcomes to working alone.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
The source backs part of the claim, not all of it — or backs it for a different population, condition or comparison.
In the citing paper
“AI agency patterns are typically co-operative or semi-active [66, 191], often involving a shared or dynamically negotiated locus of control [8, 47, 109].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [191]
Experimental — accuracy not yet measuredThe cited source says what the claim attributes to it.
In the citing paper
“Transparency and explainability regarding AI actions [70, 187, 209] are crucial for enabling humans to appropriately trust and engage with AI contributions [191, 201].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [191]
The cited source says what the claim attributes to it.
In the citing paper
“Key risks include potential job displacement via automation [191], generating inappropriate or harmful content [199], and disrupting creative economies, which may face user objections [122].”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [191]
The cited source says what the claim attributes to it.
In the citing paper
“Our analysis helped us identify several characteristics unique to generative AI that have implications on the user experience: the models' capability of producing multiple outputs [116, 135, 150], the possibility of flaws or imperfections 15 within those outputs [183, 184], and the various ways that people can control or influence those outputs [85, 103, 105].”
From Design Principles for Generative AI Applications (2024) · cites this paper as [183]
The check wasn’t confident either way, so it gives no verdict — never disputed.
In the citing paper
“Educational AI systems integrate curriculum-bound reactive and proactive mechanisms.”
From Exploring Collaboration Patterns and Strategies in Human-AI Co-creation through the Lens of Agency: A Scoping Review of the Top-tier HCI Literature (2025) · cites this paper as [191]
Experimental — accuracy not yet measured
1 more claim about this paper
Create a free account to see every claim other papers make about this one.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Language Models are Few-Shot Learners
Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020) · arXiv · doi:10.48550/arxiv.2005.14165 · Cited by 23,372
Rayward-Smith, V. J. (1991) · Journal of the Operational Research Society · doi:10.1057/jors.1991.155 · Cited by 16,821
Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research
(1988) · Advances in Psychology · doi:10.1016/s0166-4115(08)62386-9 · Cited by 15,003
Kaugeranna, R., Kaugeranna, E., & Aion, (. S. 4. 6. (2023) · DROPS (Schloss Dagstuhl – Leibniz Center for Informatics) · doi:10.4230/lipics.giscience.2023.43 · Cited by 13,690
Grossman, P. (2009) · Discrete Mathematics for Computing · doi:10.1007/978-0-230-37405-8_1 · Cited by 8,834
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · Findings of the Association for Computational Linguistics: EMNLP 2020 · doi:10.18653/v1/2020.findings-emnlp.139 · Cited by 2,824
Evaluating Large Language Models Trained on Code
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. (2021) · arXiv · doi:10.48550/arxiv.2107.03374 · Cited by 1,436
Fredkin, E. (1960) · Communications of the ACM · doi:10.1145/367390.367400 · Cited by 1,090
(2005) · Texas ScholarWorks (Texas Digital Library) · Cited by 1,064
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2018) · The Quarterly Journal of Economics · doi:10.1093/qje/qjx032 · Cited by 1,040
Unified Pre-training for Program Understanding and GenerationPreprint
Ahmad, W. U., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · North American Chapter of the Association for Computational Linguistics · doi:10.48550/arxiv.2103.06333 · Cited by 1,001
Questioning the AI: Informing Design Practices for Explainable AI User Experiences
Liao, Q. V., Gruen, D., & Miller, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376590 · Cited by 925
Machines as teammates: A research agenda on AI in team collaboration
Seeber, I., Bittner, E., Briggs, R. O., de Vreede, T., de Vreede, G.-J., Elkins, A., Maier, R., Merz, A. B., Oeste-Reiß, S., Randrup, N., et al. (2020) · Information & Management · doi:10.1016/j.im.2019.103174 · Cited by 835 · cited 2×
“” These relational descriptions evoke Lubart’s proposed roles for computer support as a “nanny,” “coach,” or “colleague” [56], as well as the “teacher” role described by Wang et al. [94], the “partner” role proposed by Main and Grierson [59], and the “teammate” role promoted by Seeber et al. [85].”
BackgroundShneiderman, B. (2020) · ACM Transactions on Interactive Intelligent Systems · doi:10.1145/3419764 · Cited by 768
Parasuraman, R., Sheridan, T. B., & Wickens, C. D. (2008) · Journal of Cognitive Engineering and Decision Making · doi:10.1518/155534308x284417 · Cited by 695
Unified Pre-training for Program Understanding and Generation
Ahmad, W., Chakraborty, S., Ray, B., & Chang, K.-W. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.211 · Cited by 642
Human Decisions and Machine Predictions
Kleinberg, J., Lakkaraju, H., Leskovec, J., Ludwig, J., & Mullainathan, S. (2017) · doi:10.3386/w23180 · Cited by 545
Expanding Explainability: Towards Social Transparency in AI systems
Ehsan, U., Liao, Q. V., Muller, M., Riedl, M. O., & Weisz, J. D. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445188 · Cited by 538 · cited 1×
“We observed participants asking (of themselves) many of the same kinds of “why” and “how” questions [54] that researchers in explainable AI (XAI) are addressing through technological [5], and even social [26], means.”
BackgroundSelf-citationThe Pile: An 800GB Dataset of Diverse Text for Language Modeling
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., et al. (2021) · arXiv · doi:10.48550/arxiv.2101.00027 · Cited by 492
GraphCodeBERT: Pre-training Code Representations with Data Flow
Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.08366 · Cited by 480
What we know about spreadsheet errors
Panko, R. R. (1998) · Journal of Organizational and End User Computing · doi:10.4018/joeuc.1998040102 · Cited by 464
Toward human-centered AI: a perspective from human-computer interaction
Xu, W. (2019) · Interactions · doi:10.1145/3328485 · Cited by 454
Human-Centered Artificial Intelligence: Three Fresh Ideas
University of Maryland, C. P., & Shneiderman, B. (2020) · AIS Transactions on Human-Computer Interaction · doi:10.17705/1thci.00131 · Cited by 421
GPT-NeoX-20B: An Open-Source Autoregressive Language Model
Black, S., Biderman, S., Hallahan, E., Anthony, Q., Gao, L., Golding, L., He, H., Leahy, C., McDonell, K., Phang, J., et al. (2022) · Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models · doi:10.18653/v1/2022.bigscience-1.9 · Cited by 404
The Principles and Limits of Algorithm-in-the-Loop Decision Making
Green, B., & Chen, Y. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359152 · Cited by 392 · cited 2×
“Decision Making Green and Chen [37] AI-supported decisions were more accurate, but not more reliable or fair Mixed outcomes”
BackgroundLai, V., & Tan, C. (2019) · Proceedings of the Conference on Fairness, Accountability, and Transparency · doi:10.1145/3287560.3287590 · Cited by 377
Human‐centered artificial intelligence and machine learning
Riedl, M. O. (2019) · Human Behavior and Emerging Technologies · doi:10.1002/hbe2.117 · Cited by 371
Lubart, T. (2005) · International Journal of Human-Computer Studies · doi:10.1016/j.ijhcs.2005.04.002 · Cited by 365
Unit Test Case Generation with Transformers and Focal ContextPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2009.05617 · Cited by 346
Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
Wang, D., Weisz, J. D., Muller, M., Ram, P., Geyer, W., Dugan, C., Tausczik, Y., Samulowitz, H., & Gray, A. (2019) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3359313 · Cited by 342
Self-citationGPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow
Black, S., Leo, G., Wang, P., Leahy, C., & Biderman, S. (2021) · Zenodo · doi:10.5281/zenodo.5297715 · Cited by 324
Wang, X., & Yin, M. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450650 · Cited by 320 · cited 2×
“Wang and Yin [95] examined the role of explanations in AI-supported decision making and found that various XAI methods were ineffective in supporting human decision makers on tasks for which they had limited domain expertise.”
On the naturalness of software
Hindle, A., Barr, E. T., Gabel, M., Su, Z., & Devanbu, P. (2016) · Communications of the ACM · doi:10.1145/2902362 · Cited by 307
Oh, C., Song, J., Choi, J., Kim, S., Lee, S., & Suh, B. (2018) · Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3173574.3174223 · Cited by 301
Information and Incentives Inside the Firm: Evidence from Loan Officer Rotation
HERTZBERG, A., LIBERTI, J. M., & PARAVISINI, D. (2010) · The Journal of Finance · doi:10.1111/j.1540-6261.2010.01553.x · Cited by 287
Human-Centered Explainable AI: Towards a Reflective Sociotechnical Approach
Ehsan, U., & Riedl, M. O. (2020) · Lecture Notes in Computer Science · doi:10.1007/978-3-030-60117-1_33 · Cited by 274
Creative Writing with a Machine in the Loop: Case Studies on Slogans and Stories
Clark, E., Ross, A. S., Tan, C., Ji, Y., & Smith, N. A. (2018) · 23rd International Conference on Intelligent User Interfaces · doi:10.1145/3172944.3172983 · Cited by 257
Learning to Generate Pseudo-Code from Source Code Using Statistical Machine Translation
Oda, Y., Fudaba, H., Neubig, G., Hata, H., Sakti, S., Toda, T., & Nakamura, S. (2015) · 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE) · doi:10.1109/ase.2015.36 · Cited by 250
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., et al. (2020) · arXiv · doi:10.48550/arxiv.2002.08155 · Cited by 243
Some psychological evidence on how people debug computer programs
Gould, J. D. (1975) · International Journal of Man-Machine Studies · doi:10.1016/s0020-7373(75)80005-8 · Cited by 237
Code Prediction by Feeding Trees to Transformers
Kim, S., Zhao, J., Tian, Y., & Chandra, S. (2021) · 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) · doi:10.1109/icse43902.2021.00026 · Cited by 209
A framework and methodology for studying the causes of software errors in programming systems
Ko, A. J., & Myers, B. A. (2005) · Journal of Visual Languages & Computing · doi:10.1016/j.jvlc.2004.08.003 · Cited by 195
Guzdial, M., Liao, N., Chen, J., Chen, S.-Y., Shah, S., Shah, V., Reno, J. O., Smith, G., & Riedl, M. O. (2019) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300854 · Cited by 176
O’Connor, A. M., Tsafnat, G., Thomas, J., Glasziou, P., Gilbert, S. B., & Hutton, B. (2019) · Systematic Reviews · doi:10.1186/s13643-019-1062-0 · Cited by 167 · cited 1×
“Building trust with automated tools is important for having users adopt those tools into their workflows [9, 68], and prior work shows that providing more information about how an automated system works can improve appropriate trust (e.”
MethodTraduction Non Supervisée de Langages de Programmation
Rozière, B. (2022) · theses.fr (ABES) · Cited by 167
Knuth, D. E. (1989) · Software Practice and Experience · doi:10.1002/spe.4380190702 · Cited by 164
In-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2022) · ACM Transactions on Software Engineering and Methodology · doi:10.1145/3487569 · Cited by 154 · cited 3×
“[102] conducted an extensive evaluation of Python programmers who used a natural language to code model to complete a variety of programming tasks.”
BackgroundMethodInfluentialIn-IDE Code Generation from Natural Language: Promise and Challenges
Xu, F. F., Vasilescu, B., & Neubig, G. (2021) · arXiv · doi:10.48550/arxiv.2101.11149 · Cited by 153
A Survey of Machine Learning for Big Code and Naturalness
Allamanis, M., Barr, E. T., Devanbu, P., & Sutton, C. (2019) · ACM Computing Surveys · doi:10.1145/3212695 · Cited by 150 · cited 1×
“This idea was promoted by the naturalness hypothesis [4, 21, 42], that software is just another form of human communication.”
MethodMetaphoria: An Algorithmic Companion for Metaphor Creation
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3290605.3300526 · Cited by 146
Automation, workers’ skills and job satisfaction
Schwabe, H., & Castellacci, F. (2020) · PLOS ONE · doi:10.1371/journal.pone.0242929 · Cited by 142
A comparative study of parallel and sequential priority queue algorithms
Rönngren, R., & Ayani, R. (1997) · ACM Transactions on Modeling and Computer Simulation · doi:10.1145/249204.249205 · Cited by 135
Does projection into use improve trust and exploration? An example with a cruise control system
Cahour, B., & Forzy, J.-F. (2009) · Safety Science · doi:10.1016/j.ssci.2009.03.015 · Cited by 134
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3476089 · Cited by 134 · cited 1×
“Although systems that abdicate all decision-making power to AI models have been strongly criticized [41, 45, 47, 57, 80, 81], studies of whether human-AI joint decision making results in improved decision quality are inconclusive.”
BackgroundVisual, textual or hybrid: the effect of user expertise on different explanations
Szymanski, M., Millecamp, M., & Verbert, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450662 · Cited by 131
Perfection Not Required? Human-AI Partnerships in Code Translation
Weisz, J. D., Muller, M., Houde, S., Richards, J., Ross, S. I., Martinez, F., Agarwal, M., & Talamadupula, K. (2021) · 26th International Conference on Intelligent User Interfaces · doi:10.1145/3397481.3450656 · Cited by 129
Self-citationDrozdal, J., Weisz, J., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M., Ju, L., & Su, H. (2020) · Proceedings of the 25th International Conference on Intelligent User Interfaces · doi:10.1145/3377325.3377501 · Cited by 119
Self-citationBuilding Trust in Artificial Intelligence
Rossi, F. (2018) · Journal of international affairs · Cited by 118
A Human-Centered Review of Algorithms used within the U.S. Child Welfare System
Saxena, D., Badillo-Urquiola, K., Wisniewski, P. J., & Guha, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376229 · Cited by 116
Drosos, I., Barik, T., Guo, P. J., DeLine, R., & Gulwani, S. (2020) · Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3313831.3376442 · Cited by 107
CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Tasks
Puri, R., Kung, D. S., Janssen, G., Zhang, W., Domeniconi, G., Zolotov, V., Dolby, J., Chen, J., Choudhury, M., Decker, L., et al. (2021) · arXiv · doi:10.48550/arxiv.2105.12655 · Cited by 92
Trie: An alternative data structure for data mining algorithms
Bodon, F., & Rónyai, L. (2003) · Mathematical and Computer Modelling · doi:10.1016/0895-7177(03)90058-6 · Cited by 91
Increasing the Speed and Accuracy of Data Labeling Through an AI Assisted Interface
Desmond, M., Muller, M. J., Ashktorab, Z., Dugan, C., Duesterwald, E., Brimijoin, K., Finegan‐Dollak, C., Brachman, M., Sharma, A., Joshi, N. N., et al. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450698 · Cited by 78
Self-citationMigrating code with statistical machine translation
Nguyen, A. T., Nguyen, T. T., & Nguyen, T. N. (2014) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/2591062.2591072 · Cited by 67
Perceptions about the impact of automation in the workplace
Dodel, M., & Mesch, G. S. (2020) · Information Communication & Society · doi:10.1080/1369118x.2020.1716043 · Cited by 64
collabdraw: An Environment for Collaborative Sketching with an Artificial Agent
Fan, J. E., Dinculescu, M., & Ha, D. (2019) · Creativity and Cognition · doi:10.1145/3325480.3326578 · Cited by 57
Unsupervised translation of programming languages
Roziere, B. (2026) · doi:10.70675/416f6582z4576z444eza710z5e930d59bf97 · Cited by 57
An instructional scaffolding approach to teaching software design
Linder, S. P., Abbott, D., & Fromberger, M. J. (2006) · doi:10.5555/1127442.1127472 · Cited by 54
AI Explainability 360: An Extensible Toolkit for Understanding Data and Machine Learning Models
Arya, V., Bellamy, R., Chen, P., Dhurandhar, A., Hind, M., Hoffman, S. C., Houde, S., Liao, Q. V., Luss, R., Mojsilović, A., et al. (2020) · Journal of Machine Learning Research · Cited by 54
Self-citationIncreasing the User Experience in Autonomous Driving through different Feedback Modalities
Schneider, T., Ghellal, S., Love, S., & Gerlicher, A. R. S. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450687 · Cited by 51
AVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2023) · Findings of the Association for Computational Linguistics: ACL 2023 · doi:10.18653/v1/2023.findings-acl.143 · Cited by 46
AVATAR: A Parallel Corpus for Java-Python Program Translation
Ahmad, W. U., Tushar, M. G. R., Chakraborty, S., & Chang, K.-W. (2021) · arXiv · doi:10.48550/arxiv.2108.11590 · Cited by 46 · cited 2×
“Neural machine translation (NMT) models trained in this way, coupled with large code datasets (e.g. Project CodeNet [72], AVATAR [2]), are making it easier to create code translation systems across a wide range of languages.”
BackgroundAI-Assisted Human Labeling: Batching for Efficiency without Overreliance
Ashktorab, Z., Desmond, M., Andres, J., Muller, M., Joshi, N. N., Brachman, M., Sharma, A., Brimijoin, K., Pan, Q., Wolf, C. T., et al. (2021) · Proceedings of the ACM on Human-Computer Interaction · doi:10.1145/3449163 · Cited by 45
Self-citationKuttal, S. K., Ong, B., Kwasny, K., & Robe, P. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445659 · Cited by 44
Mixed-Initiative Level Design with RL Brush
Delarosa, O., Dong, H., Ruan, M., Khalifa, A., & Togelius, J. (2021) · Lecture notes in computer science · doi:10.1007/978-3-030-72914-1_27 · Cited by 41
Evaluating the Interpretability of Generative Models by Interactive Reconstruction
Ross, A., Chen, N., Hang, E. Z., Glassman, E. L., & Doshi-Velez, F. (2021) · Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445296 · Cited by 40
(2021) · arXiv (Cornell University) · Cited by 37
Warner, J., & Guo, P. J. (2017) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3025453.3025876 · Cited by 35
How a Stylistic, Machine-Generated Thesaurus Impacts a Writer's Process
Gero, K. I., & Chilton, L. B. (2019) · Proceedings of the 2019 on Creativity and Cognition · doi:10.1145/3325480.3326573 · Cited by 32
Trust in Collaborative Automation in High Stakes Software Engineering Work: A Case Study at NASA
Widder, D. G., Dabbish, L., Herbsleb, J. D., Holloway, A., & Davidoff, S. (2021) · CHI Conference on Human Factors in Computing Systems · doi:10.1145/3411764.3445650 · Cited by 31
Draw with me: human-in-the-loop for image restoration
Weber, T., Hußmann, H., Han, Z., Matthes, S., & Liu, Y. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3377325.3377509 · Cited by 29
Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
Narkar, S., Zhang, Y., Liao, Q. V., Wang, D., & Weisz, J. D. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450658 · Cited by 29
Self-citationWang, D., Ram, P., Weidele, D. K. I., Liu, S., Muller, M. J., Weisz, J. D., Valente, A. N., Chaudhary, A., Torres, D., Samulowitz, H., et al. (2020) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3379336.3381474 · Cited by 28
Self-citationDrozdal, J., Weisz, J. D., Wang, D., Dass, G., Yao, B., Zhao, C., Muller, M. J., Lin, J., & Su, H. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2001.06509 · Cited by 25
Self-citationJones, B. H., King, P. T., Baker, G., & Ingham, T. R. (2020) · American Indian Culture and Research Journal · doi:10.17953/aicrj.44.2.jones · Cited by 24
(Mis)Attributing the Causes of American Job Loss
Mutz, D. C. (2021) · Public Opinion Quarterly · doi:10.1093/poq/nfab003 · Cited by 24
Unit Test Case Generation with TransformersPreprint
Tufano, M., Drain, D., Svyatkovskiy, A., Deng, S. K., & Sundaresan, N. (2020) · arXiv (Cornell University) · Cited by 23
Respecting Human Autonomy through Human-Centered AI
Sankaran, S., Zhang, C., Lopez, M. G., & Väänänen, K. (2020) · Nordic Conference on Human-Computer Interaction · doi:10.1145/3419249.3420098 · Cited by 22
Main, A., & Grierson, M. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2007.04848 · Cited by 20
Quantifying Compliance and Reliance Trust Behaviors to Influence Trust in Human-Automation Teams
Boubin, J. G., Rusnock, C. F., & Bindewald, J. M. (2017) · Proceedings of the Human Factors and Ergonomics Society Annual Meeting · doi:10.1177/1541931213601672 · Cited by 17
Tutorial: Human-Centered AI: Reliable, Safe and Trustworthy
Shneiderman, B. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397482.3453994 · Cited by 16
Technology-induced job loss risk, disability and all-cause mortality in Norway
Bratsberg, B., Røgeberg, O. J., & Skirbekk, V. (2021) · Occupational and Environmental Medicine · doi:10.1136/oemed-2021-107598 · Cited by 16
Wolf, C. T. (2020) · XRDS Crossroads The ACM Magazine for Students · doi:10.1145/3398370 · Cited by 14
Eisenberg, M., & Peelle, H. A. (1983) · ACM SIGAPL APL Quote Quad · doi:10.1145/390005.801189 · Cited by 10
Technology scenario: employment implications of radical automation
Lewney, R. (2019) · Cited by 10
New Initiative: The Naturalness of Software
Devanbu, P. (2015) · 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering · doi:10.1109/icse.2015.190 · Cited by 9
Model-Based Scaffolding Code Generation for Cross-Platform Applications
Inayatullah, M., Azam, F., & Anwar, M. W. (2019) · IEEE Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON) · doi:10.1109/iemcon.2019.8936289 · Cited by 8
Themisto: Towards Automated Documentation Generation in Computational Notebooks.Preprint
Wang, A. Y., Wang, D., Drozdal, J., Muller, M. J., Park, S., Weisz, J. D., Liu, X., Wu, L., & Dugan, C. (2021) · arXiv (Cornell University) · Cited by 8
Self-citationDesigning for Democratization: Introducing Novices to Artificial Intelligence Via Maker KitsPreprint
Dibia, V. C., Cox, A., & Weisz, J. D. (2018) · arXiv (Cornell University) · doi:10.48550/arxiv.1805.10723 · Cited by 7
Self-citationDraw with Me: Human-in-the-Loop for Image Restoration
Weber, T., Han, Z., Matthes, S., Hußmann, H., & Liu, Y. (2020) · Lecture notes in computer science · doi:10.1007/978-3-030-58285-2_19 · Cited by 7
Kizilaslan, A., & Lookman, A. A. (2017) · SSRN Electronic Journal · doi:10.2139/ssrn.2987613 · Cited by 6
Wiehr, F., Cakar, B., Daiber, F., & Krüger, A. (2021) · ACM International Conference on Intelligent User Interfaces (IUI) · doi:10.1145/3397481.3450677 · Cited by 6
Towards democratizing computer science education through social game design
Ahmadi, N., Jazayeri, M., & Repenning, A. (2011) · IEEE/ACM International Workshop on Games and Software Engineering (GAS) · doi:10.1145/1984674.1984691 · Cited by 5
Lyn, A. (2021) · SSRN Electronic Journal · doi:10.2139/ssrn.3831441 · Cited by 4
Research works printed but not linked: 9
- Stephen Cass . 2021. Top Programming Languages 2021 . IEEE Spectrum ( 24 August 2021). https://spectrum.ieee.org/top-programming-languages/ Stephen Cass. 2021. Top Programming Languages 2021. IEEE Spectrum (24 August 2021). https://spectrum.ieee.org/top-programming-languages/
- Sam Corbett-Davies , Sharad Goel , and Sandra González-Bailón . 2017. Even imperfect algorithms can improve the criminal justice system. New York Times ( 2017 ). Sam Corbett-Davies, Sharad Goel, and Sandra González-Bailón. 2017. Even imperfect algorithms can improve the criminal justice system. New York Times (2017).
- Marc Eisenstadt . 1993 . Tales of debugging from the front lines . In Empirical Studies of Programmers: Fifth Workshop. Palo Alto, CA: Ablex Publishing Corporation, 86–112 . Marc Eisenstadt. 1993. Tales of debugging from the front lines. In Empirical Studies of Programmers: Fifth Workshop. Palo Alto, CA: Ablex Publishing Corporation, 86–112.
- W Lewis Johnson , Elliot Soloway , Benjamin Cutler , and Steven Draper . 1983. Bug catalogue: I . Yale University Press . W Lewis Johnson, Elliot Soloway, Benjamin Cutler, and Steven Draper. 1983. Bug catalogue: I. Yale University Press.
- Jeremy Miles and Mark Shevlin . 2001. Applying regression and correlation: A guide for students and researchers . Sage . Jeremy Miles and Mark Shevlin. 2001. Applying regression and correlation: A guide for students and researchers. Sage.
- Alex P Miller . 2018. Want less-biased decisions? Use algorithms. Harvard business review 26 ( 2018 ). Alex P Miller. 2018. Want less-biased decisions? Use algorithms. Harvard business review 26 (2018).
- 10.5555/287893.287899 · doi:10.5555/287893.287899
- Alec Radford , Jeffrey Wu , Rewon Child , David Luan , Dario Amodei , Ilya Sutskever , 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 ( 2019 ), 9. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9.
- Douglas Schuler and Aki Namioka . 1993. Participatory design: Principles and practices . CRC Press . Douglas Schuler and Aki Namioka. 1993. Participatory design: Principles and practices. CRC Press.
Cited by
The Metacognitive Demands and Opportunities of Generative AI
Tankelevitch, L., Kewenig, V., Simkute, A., Scott, A. E., Sarkar, A., Sellen, A., & Rintel, S. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642902 · Cited by 493
Lost in Translation: A Study of Bugs Introduced by Large Language Models while Translating Code
Pan, R., Ibrahimzada, A. R., Krishna, R., Sankar, D., Wassi, L. P., Merler, M., Sobolev, B., Pavuluri, R., Sinha, S., & Jabbarvand, R. (2024) · Proceedings of the IEEE/ACM 46th International Conference on Software Engineering · doi:10.1145/3597503.3639226 · Cited by 349
Ross, S. I., Martinez, F., Houde, S., Muller, M., & Weisz, J. D. (2023) · Proceedings of the 28th International Conference on Intelligent User Interfaces · doi:10.1145/3581641.3584037 · Cited by 258
Self-citationDesign Principles for Generative AI Applications
Weisz, J. D., He, J., Muller, M., Hoefer, G., Miles, R., & Geyer, W. (2024) · Proceedings of the CHI Conference on Human Factors in Computing Systems · doi:10.1145/3613904.3642466 · Cited by 232
“…to generative AI that have implications on the user experience: the models’ capability of producing multiple outputs [116, 135, 150], the possibility of faws or imperfections 15 within those outputs [183, 184], and the various ways that people can control or infuence those outputs [85, 103, 105].”
BackgroundMethodSelf-citationWhat is Human-Centered about Human-Centered AI? A Map of the Research Landscape
Capel, T., & Brereton, M. (2023) · Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems · doi:10.1145/3544548.3580959 · Cited by 228
“…combine human meaning interpretation together with AI techniques e.g., fact checking [205], journalism [289], unstructured text analysis [270], code documentation [299], translating source code into diferent programming languages [308], and creation of presentation slides for data scientists [322].”
Background
Metadata
- Authors
- Weisz, J. D., Muller, M., Ross, S. I., Martinez, F., Houde, S., Agarwal, M., Talamadupula, K., & Richards, J. T.
- Affiliations
- IBM (United States) (US)
- ORCID
- Venue
- 27th International Conference on Intelligent User Interfaces
- Publisher
- ACM
- Published
- 2022-03-22
- Pages
- 369-391
- OpenAlex
- W4221015471
- Semantic Scholar
- 7438626a757c5442b9c0fb37b54ec0fe7e1889c3
- Cited by
- 69
- References
- 95
- Citations by year
- 2026: 6 · 2025: 25 · 2024: 26 · 2023: 10 · 2022: 2
- FWCI
- 14.08
- Type
- conference paper
- Language
- en
- Open access
- green
- arxiv.org
- Locations
- ACM International Conference on Intelligent User Interfaces (IUI) (publishedVersion)
- arXiv (Cornell University) (submittedVersion)
- arXiv (Cornell University)
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Information Systems › Software Engineering Techniques and Practices
- Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
- Keywords
- program translation, generative AI, programming language translation, human-AI collaboration, AI-generated code quality, multiple translations, intelligent user interfaces, code errors
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.