CoChat
Sign Up
PreprintCited by 69Teaching and Learning ProgrammingGenetics, Bioinformatics, and Biomedical ResearchArtificial Intelligence in Education

Many bioinformatics programming tasks can be automated with ChatGPT

Open access
Piccolo, Stephen R., Denny, Paul, Luxton-Reilly, Andrew, Payne, Samuel, Ridge, Perry G.·2023·arXiv
DOI 10.48550/arxiv.2303.13528arXiv 2303.13528PMC PMC10564134
Open this paper in CoChat — free account

Abstract

Computer programming is a fundamental tool for life scientists, allowing them to carry out many essential research tasks. However, despite a variety of educational efforts, learning to write code can be a challenging endeavor for both researchers and students in life science disciplines. Recent advances in artificial intelligence have made it possible to translate human-language prompts to functional code, raising questions about whether these technologies can aid (or replace) life scientists' efforts to write code. Using 184 programming exercises from an introductory-bioinformatics course, we evaluated the extent to which one such model -- OpenAI's ChatGPT -- can successfully complete basic- to moderate-level programming tasks. On its first attempt, ChatGPT solved 139 (75.5%) of the exercises. For the remaining exercises, we provided natural-language feedback to the model, prompting it to try different approaches. Within 7 or fewer attempts, ChatGPT solved 179 (97.3%) of the exercises. These findings have important implications for life-sciences research and education. For many programming tasks, researchers no longer need to write code from scratch. Instead, machine-learning models may produce usable solutions. Instructors may need to adapt their pedagogical approaches and assessment techniques to account for these new capabilities that are available to the general public.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2303.13528

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • LLMs have demonstrated remarkable effectiveness in generating novel language artifacts such as course readings, code explanations, model solutions, feedback, and responses to help requests.

    “This has been inspired by many other instances where LLMs have already demonstrated remarkable effectiveness in generating novel language artifacts. These include course readings [22], code explanations [23, 28, 29, 41], model solutions [9, 35, 42], feedback [34], or responses to help requests [18, 26, 46, 47].”

    From A Comparative Study of AI-Generated (GPT-4) and Human-crafted MCQs in Programming Education (2024) · cites this paper as [35]

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 28

  1. [5]5. Guzdial, M. Computational thinking and using programming to learn. in Learner-centered design of computing education: Research on computing for everyone 37-51 (Springer International Publishing, 2016). doi:10.1007/978-3-031-02216-6\_3. · DOI
  2. [14]14. Prabhu, P. et al. A survey of the practice of computational science. in State Pract. Rep. 1-12 (Association for Computing Machinery, 2011). doi:10.1145/2063348.2063374. · DOI
  3. [17]17. Dasu, T. & Johnson, T. Exploratory data mining and data cleaning . (John Wiley & Sons, 2003).
  4. [18]18. Simon, Mason, R., Crick, T., Davenport, J. H. & Murphy, E. Language Choice in Introductory Programming Courses at Australasian and UK Universities. in Proc. 49th ACM Tech. Symp. Comput. Sci. Educ. 852-857 (Association for Computing Machinery, 2018). doi:10.1145/3159450.3159547. · DOI
  5. [21]21. McKinney, W. Data Structures for Statistical Computing in Python. in Proc. 9th Python Sci. Conf. 6 (2010).
  6. [24]24. Géron, A. Hands-On Machine Learning with Scikit-Learn and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems . (O'Reilly Media, Inc., 2017).
  7. [26]26. Kelleher, C. & Pausch, R. Lowering the barriers to programming: A taxonomy of programming environments and languages for novice programmers. ACM Comput. Surv. 37 , 83-137 (2005).
  8. [28]28. Luxton-Reilly, A. et al. Introductory programming: A systematic literature review. in Proc. Companion 23rd Annu. ACM Conf. Innov. Technol. Comput. Sci. Educ. 55-106 (2018).
  9. [29]29. Smith, R. & Rixner, S. The Error Landscape: Characterizing the Mistakes of Novice Programmers. in Proc. 50th ACM Tech. Symp. Comput. Sci. Educ. 538-544 (Association for Computing Machinery, 2019). doi:10.1145/3287324.3287394. · DOI
  10. [30]30. Becker, B. A. et al. Compiler Error Messages Considered Unhelpful: The Landscape of TextBased Programming Error Message Research. in Proc. Work. Group Rep. Innov. Technol. Comput. Sci. Educ. 177-210 (Association for Computing Machinery, 2019). doi:10.1145/3344429.3372508. · DOI · Paper page
  11. [31]31. Finnie-Ansley, J., Denny, P., Becker, B. A., Luxton-Reilly, A. & Prather, J. The Robots Are Coming: Exploring the Implications of OpenAI Codex on Introductory Programming. in Proc. 24th Australas. Comput. Educ. Conf. 10-19 (Association for Computing Machinery, 2022). doi:10.1145/3511861.3511863. · DOI · Paper page
  12. [32]32. Finnie-Ansley, J. et al. My AI Wants to Know if This Will Be on the Exam: Testing OpenAI’s Codex on CS2 Programming Exercises. in Proc. 25th Australas. Comput. Educ. Conf. 97-104 (Association for Computing Machinery, 2023). doi:10.1145/3576123.3576134. · DOI · Paper page
  13. [33]33. Chen, M. et al. Evaluating Large Language Models Trained on Code. (2021) doi:10.48550/arXiv.2107.03374. · DOI · Paper page
  14. [40]40. Jiao, W., Wang, W., Huang, J., Wang, X. & Tu, Z. Is ChatGPT A Good Translator? A Preliminary Study. (2023) doi:10.48550/arXiv.2301.08745. · DOI · Paper page
  15. [42]42. Jalil, S., Rafi, S., LaToza, T. D., Moran, K. & Lam, W. ChatGPT and Software Testing Education: Promises & Perils. (2023) doi:10.48550/arXiv.2302.03287. · DOI
  16. [57]57. Owens, B. How Nature readers are using ChatGPT. Nature (2023) doi:10.1038/d41586-02300500-8. · DOI
  17. [58]58. Austin, J. et al. Program Synthesis with Large Language Models. (2021) doi:10.48550/arXiv.2108.07732. · DOI · Paper page
  18. [59]59. Simon, H. A. Experiments with a Heuristic Compiler. J. ACM 10 , 493-506 (1963).
  19. [60]60. Manna, Z. & Waldinger, R. J. Toward automatic program synthesis. Commun. ACM 14 , 151-165 (1971).
  20. [61]61. Feng, Z. et al. Codebert: A pre-trained model for programming and natural languages. ArXiv Prepr. ArXiv200208155 (2020).
  21. [62]62. Clement, C. B., Drain, D., Timcheck, J., Svyatkovskiy, A. & Sundaresan, N. PyMT5: Multi-mode translation of natural language and Python code with transformers. ArXiv Prepr. ArXiv201003150 (2020).
  22. [63]63. Denny, P., Kumar, V. & Giacaman, N. Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language. in Proc. 54th ACM Tech. Symp. Comput. Sci. Educ. V 1 1136-1142 (Association for Computing Machinery, 2023). doi:10.1145/3545945.3569823. · DOI · Paper page
  23. [64]64. Yellin, D. M. The Premature Obituary of Programming. Commun. ACM 66 , 41-44 (2023).
  24. [66]66. Denny, P., Luxton-Reilly, A., Craig, M. & Petersen, A. Improving complex task performance using a sequence of simple practice tasks. in Proc. 23rd Annu. ACM Conf. Innov. Technol. Comput. Sci. Educ. 4-9 (Association for Computing Machinery, 2018). doi:10.1145/3197091.3197141. · DOI
  25. [68]68. Lehtinen, T., Haaranen, L. & Leinonen, J. Automated Questionnaires About Students’ JavaScript Programs: Towards Gauging Novice Programming Processes. in Proc. 25th Australas. Comput. Educ. Conf. 49-58 (Association for Computing Machinery, 2023). doi:10.1145/3576123.3576129. · DOI · Paper page
  26. [69]69. Crow, T., Luxton-Reilly, A. & Wuensche, B. Intelligent tutoring systems for programming education: A systematic review. in Proc. 20th Australas. Comput. Educ. Conf. 53-62 (2018).
  27. [70]70. MacNeil, S. et al. Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book. in Proc. 54th ACM Tech. Symp. Comput. Sci. Educ. V 1 931-937 (Association for Computing Machinery, 2023). doi:10.1145/3545945.3569785. · DOI · Paper page
  28. [71]71. Sarsa, S., Denny, P., Hellas, A. & Leinonen, J. Automatic Generation of Programming Exercises and Code Explanations Using Large Language Models. in Proc. 2022 ACM Conf. Int. Comput. Educ. Res. - Vol. 1 vol. 1 27-43 (Association for Computing Machinery, 2022).

Cited by

Sorted by citations

Authors
Piccolo, S. R., Denny, P., Luxton-Reilly, A., Payne, S., & Ridge, P. G.
Affiliations
  • Brigham Young University (US)
  • University of Auckland (NZ)
Corresponding author
Piccolo, Stephen R.
ORCID
Venue
arXiv
Publisher
arXiv
Published
2023
Volume
19
Issue
9
Pages
e1011511-e1011511
ISSN
1553-734X (ISSN-L)
DOI
10.48550/arxiv.2303.13528
arXiv
2303.13528
PMC
PMC10564134
OpenAlex
W4387129749
Cited by
69
References
0
Citations by year
2026: 9 · 2025: 17 · 2024: 29 · 2023: 14
FWCI
15.68
Type
journal article
Language
en
Open access
Licence
Locations
  • PLoS Computational Biology (publishedVersion, cc-by)
  • PubMed Central (publishedVersion, cc-by)
  • PubMed (publishedVersion)
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • ScholarsArchive (Brigham Young University) (submittedVersion, cc-by)
  • DOAJ (DOAJ: Directory of Open Access Journals) (submittedVersion, cc-by-sa)
  • arXiv (Cornell University) (cc-by)
Topics
  • Physical Sciences › Computer Science › Computer Science Applications › Teaching and Learning Programming
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genetics, Bioinformatics, and Biomedical Research
  • Physical Sciences › Computer Science › Information Systems › Artificial Intelligence in Education
Keywords
large language models, ChatGPT, programming education, bioinformatics education, programming exercises, natural language programming, life sciences education, immediate feedback
MeSH terms
Large Language Models, Artificial Intelligence, Curriculum, Humans, Programming Languages, Software, Computational Biology

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review