BioCoder: a benchmark for bioinformatics code generation with large language models
Open accessAbstract
Summary: Pretrained large language models (LLMs) have significantly improved code generation. As these models scale up, there is an increasing need for the output to handle more intricate tasks and to be appropriately specialized to particular domains. Here, we target bioinformatics due to the amount of domain knowledge, algorithms, and data operations this discipline requires. We present BioCoder, a benchmark developed to evaluate LLMs in generating bioinformatics-specific code. BioCoder spans much of the field, covering cross-file dependencies, class declarations, and global variables. It incorporates 1026 Python functions and 1243 Java methods extracted from GitHub, along with 253 examples from the Rosalind Project, all pertaining to bioinformatics. Using topic modeling, we show that the overall coverage of the included code is representative of the full spectrum of bioinformatics calculations. BioCoder incorporates a fuzz-testing framework for evaluation. We have applied it to evaluate various models including InCoder, CodeGen, CodeGen2, SantaCoder, StarCoder, StarCoder+, InstructCodeT5+, GPT-3.5, and GPT-4. Furthermore, we fine-tuned one model (StarCoder), demonstrating that our training dataset can enhance the performance on our testing benchmark (by >15% in terms of Pass@K under certain prompt configurations and always >3%). The results highlight two key aspects of successful models: (i) Successful models accommodate a long prompt (>2600 tokens) with full context, including functional dependencies. (ii) They contain domain-specific knowledge of bioinformatics, beyond just general coding capability. This is evident from the performance gain of GPT-3.5/4 compared to the smaller models on our benchmark (50% versus up to 25%). Availability and implementation: All datasets, benchmark, Docker images, and scripts required for testing are available at: https://github.com/gersteinlab/biocoder and https://biocoder-benchmark.github.io/.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Existing code-generation benchmarks such as SWE-Bench, BioCoder-Py, ML-Bench, and MLAgentBench only consist of tasks found on secondary sources such as GitHub and Kaggle.
“Yet, these benchmarks only consist of tasks found on secondary sources, such as GitHub and Kaggle.”
From ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery (2024) · cites this paper as Tang et al., 2024c
Section 3.1 says BioCoder’s dataset originated from bioinformatics-related GitHub repositories, and Section 3.4 identifies BioCoder-Py as part of the benchmark. The paper does not establish that SWE-Bench, ML-Bench, or MLAgentBench use only tasks from GitHub, Kaggle, or other secondary sources; nor does its account of BioCoder establish the word “only.” (LLM, conf=0.82)
Experimental — accuracy not yet measured
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Competition-level code generation with AlphaCode
Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., et al. (2022) · Science · doi:10.1126/science.abq1158 · Cited by 1,043 · cited 1× · see citing works
“In response, execution-based evaluation approaches have become more prevalent (Chen et al. 2021, Khlaaf et al. 2022, Lai et al. 2022, Li et al. 2022, Wang et al. 2022b, Athiwaratkun et al. 2023).”
Self-citationGrounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2023) · Proceedings of the ACM on Programming Languages · doi:10.1145/3586030 · Cited by 436 · cited 1× · see citing works
“Large language models (LLMs) have demonstrated great success in code generation (Chen et al. 2021, 2023, Chowdhery et al. 2022, Barke et al. 2023, Li et al. 2023).”
An innovative approach for testing bioinformatics programs using metamorphic testing
Chen, T. Y., Ho, J. W. K., Liu, H., & Xie, X. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-24 · Cited by 159 · cited 1× · see citing works
“Our testing strategy is a hybrid of unit testing and fuzz testing methods, which shares similarities with the metamorphic testing methodology described in Chen et al. (2009).”
Natural Language Generation and Understanding of Big Code for AI-Assisted Programming: A Review
Wong, M.-F., Guo, S., Hang, C. N., Ho, S., & Tan, C. W. (2023) · Entropy · doi:10.3390/e25060888 · Cited by 137 · cited 2× · see citing works
“As a result, the field of code generation has seen a burgeoning number of execution-based benchmarks (Table 1) (Lee et al. 2023, Pan et al. 2023, Wong et al. 2023, Yuan et al. 2023, Zan et al. 2023), each presenting unique properties in terms of size, language coverage (Orlanski et al. 2023),…”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps-Costin, L., Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. (2023) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2023.3267446 · Cited by 109 · cited 1× · see citing works
“We used the Pass@K metric to measure the functional accuracy (Chen et al. 2021, 2022, Cassano et al. 2023) of code generation models.”
A large-scale analysis of bioinformatics code on GitHub
Russell, P. H., Johnson, R. L., Ananthan, S., Harnke, B., & Carlson, N. E. (2018) · PLoS ONE · doi:10.1371/journal.pone.0205898 · Cited by 55 · see citing works
Grounded Copilot: How Programmers Interact with Code-Generating Models
Barke, S., James, M. B., & Polikarpova, N. (2022) · arXiv · doi:10.48550/arxiv.2206.15000 · Cited by 23 · cited 1× · see citing works
“Large language models (LLMs) have demonstrated great success in code generation (Chen et al. 2021, 2023, Chowdhery et al. 2022, Barke et al. 2023, Li et al. 2023).”
Measuring The Impact Of Programming Language Distribution
Orlanski, G., Xiao, K., Garcia, X., Hui, J., Howland, J., Malmaud, J., Austin, J., Singh, R., & Catasta, M. (2023) · arXiv · doi:10.48550/arxiv.2302.01973 · cited 1×
“…benchmarks (Table 1) (Lee et al. 2023, Pan et al. 2023, Wong et al. 2023, Yuan et al. 2023, Zan et al. 2023), each presenting unique properties in terms of size, language coverage (Orlanski et al. 2023), complexity (Du et al. 2023, Zhuo 2023), and practical applicability (Yu et al. 2023).”
Flagged first, then by citations
Research works printed but not linked: 55
- Allal LB, Li R, Kocetkov D et al Santacoder: don’t reach for the stars!. arXiv preprint arXiv:2301.03988. 2023. · arXiv
- Askell A, Bai Y, Chen A et al A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. 2021. · arXiv
- Athiwaratkun B, Gouda SK, Wang Z et al Multi-lingual evaluation of code generation models. arXiv preprint arXiv:2210.14868. 2022. · arXiv
- Austin J, Odena A, Nye M et al Program synthesis with large language models. arXiv preprint arXiv:2108.07732. 2021. · arXiv
- Bai Y, Jones A, Ndousse K et al Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. 2022. · arXiv
- Biderman S, Schoelkopf H, Anthony Q et al Pythia: A suite for analyzing large language models across training and scaling. In: International Conference on Machine Learning 2023 (pp. 2397-2430). PMLR.
- Bommasani R, Hudson DA, Adeli E et al On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258. 2021. · arXiv
- Brown TB, Mann B, Ryder N et al Language models are few-shot learners. Advances in Neural Information Processing Systems. 2020;33: 1877– 901.
- Bui ND, Le H, Wang Y et al Codetf: One-stop transformer library for state-of-the-art code llm. arXiv preprint arXiv:2306.00029. 2023. · arXiv
- Chen B, Zhang F, Nguyen A et al CodeT: Code Generation with Generated Tests. In: The Eleventh International Conference on Learning Representations 2022.
- Chen M, Tworek J, Jun H et al Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. 2021. · arXiv
- Chen X, Lin M, Schärli N et al Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128. 2023. · arXiv
- Chowdhery A, Narang S, Devlin J et al Palm: Scaling language modeling with pathways. Journal of Machine Learning Research. 2023;24(240): 1– 13.
- Christopoulou F, Lampouras G, Gritta M et al Pangu-coder: Program synthesis with function-level language modeling. arXiv preprint arXiv:2207.11280. 2022. · arXiv
- Du X, Liu M, Wang K et al Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. 2023. · arXiv
- Fried D, Aghajanyan A, Lin J et al InCoder: A Generative Model for Code Infilling and Synthesis. In: The Eleventh International Conference on Learning Representations 2022.
- Hendrycks D, Basart S, Kadavath S et al Measuring Coding Challenge Competence With APPS. In: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. 2021.
- Hoffmann J, Borgeaud S, Mensch A et al Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. 2022. · arXiv
- Khlaaf H, Mishkin P, Achiam J et al A hazard analysis framework for code synthesis large language models. arXiv preprint arXiv:2207.14157. 2022. · arXiv
- Lai Y, Li C, Wang Y et al DS-1000: A natural and reliable benchmark for data science code generation. In: International Conference on Machine Learning 2023 (pp. 18319-18345). PMLR.
- Lee T, Hong S, Ahn J et al Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060. 2023. · arXiv
- Li R, Allal LB, Zi Y et al StarCoder: may the source be with you!. arXiv preprint arXiv:2305.06161. 2023. · arXiv
- Liu T, Xu C, McAuley J. 2023. RepoBench: Benchmarking Repository-Level Code Auto-Completion Systems. In: The Twelfth International Conference on Learning Representations 2023.
- Luo Z, Xu C, Zhao P et al WizardCoder: Empowering Code Large Language Models with Evol-Instruct. In: The Twelfth International Conference on Learning Representations 2023.
- Mikolov T, Chen K, Corrado G et al Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. 2013. · arXiv
- Mitchell M, Luccioni AS, Lambert N et al Measuring data. arXiv preprint arXiv:2212.05129. 2022. · arXiv
- Muennighoff N, Wang T, Sutawika L et al Crosslingual Generalization through Multitask Finetuning. In: The 61st Annual Meeting Of The Association For Computational Linguistics 2023.
- Muennighoff N, Liu Q, Zebaze A et al OctoPack: Instruction Tuning Code Large Language Models. In: The Twelfth International Conference on Learning Representations 2024.
- Nijkamp E, Pang B, Hayashi H et al CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. In: The Eleventh International Conference on Learning Representations 2022.
- Nijkamp E, Hayashi H, Xiong C et al Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint arXiv:2305.02309. 2023. · arXiv
- Olausson TX, Inala JP, Wang C et al Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09896. 2023. · arXiv
- OpenAI 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774. 2023. · arXiv
- Pan R, Ibrahimzada AR, Krishna R et al Understanding the effectiveness of large language models in code translation. arXiv preprint arXiv:2308.03109. 2023. · arXiv
- Patil SG, Zhang T, Wang X et al Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334. 2023. · arXiv
- Qin Y, Liang S, Ye Y et al Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789. 2023. · arXiv
- Radford A, Wu J, Child R et al Language models are unsupervised multitask learners. OpenAI Blog 2019; 1: 9.
- Shen B, Zhang J, Chen T et al Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936. 2023. · arXiv
- Shrivastava D, Kocetkov D, de Vries H et al RepoFusion: Training Code Models to Understand Your Repository. arXiv preprint arXiv:2306.10998. 2023. · arXiv
- Thoppilan R, Freitas DD, Hall J et al Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239. 2022. · arXiv
- Touvron H, Lavril T, Izacard G et al Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. 2023. · arXiv
- Wang S, Li Z, Qian H et al ReCode: Robustness Evaluation of Code Generation Models. In: The 61st Annual Meeting Of The Association For Computational Linguistics 2023
- Wang Y, Wang W, Joty S et al CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In: Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing 2021 (pp. 8696-8708).
- Wang Y, Le H, Gotmare AD et al CodeT5+: Open Code Large Language Models for Code Understanding and Generation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing 2023 (pp. 1069-1088).
- Wang Z, Zhou S, Fried D et al Execution-Based Evaluation for Open-Domain Code Generation. In: Findings of the Association for Computational Linguistics: EMNLP 2023 2023 (pp. 1271-1290).
- Wang Z, Cuenca G, Zhou S et al MCoNaLa: A Benchmark for Code Generation from Multiple Natural Languages. In: Findings of the Association for Computational Linguistics: EACL 2023 2023 (pp. 265-273).
- Xu Q, Hong F, Li B et al On the Tool Manipulation Capability of Open-sourced Large Language Models. In: NeurIPS 2023 Foundation Models for Decision Making Workshop 2023.
- Yu H, Shen B, Ran D et al Codereval: A benchmark of pragmatic code generation with generative pre-trained models. In: Proceedings of the 46th IEEE/ACM International Conference on Software Engineering 2024 (pp. 1-12).
- Yuan Z, Liu J, Zi Q et al Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240. 2023. · arXiv
- Zan D, Chen B, Yang D et al CERT: continual pre-training on sketches for library-oriented code generation. arXiv preprint arXiv:2206.06888. 2022. · arXiv
- Zan D, Chen B, Lin Z et al When Language Model Meets Private Library. In: Findings of the Association for Computational Linguistics: EMNLP 2022 2022b (pp. 277-288).
- Zan D, Chen B, Zhang F et al Large Language Models Meet NL2Code: A Survey. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2023 (pp. 7443-7464).
- Zeng A, Liu X, Du Z et al GLM-130B: An Open Bilingual Pre-trained Model. In: The Eleventh International Conference on Learning Representations 2022.
- Zhang SD, Tigges C, Biderman S et alCan Transformers Learn to Solve Problems Recursively?. arXiv preprint arXiv:2305.14699. 2023. · arXiv
- Zheng Q, Xia X, Zou X et al Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. arXiv preprint arXiv:2303.17568. 2023. · arXiv
- Zhuo TY. Large language models are state-of-the-art evaluators of code generation. arXiv preprint arXiv:2304.14317. 2023. · arXiv
Cited by
Sorted by citations
Benchmarking large language models for genomic knowledge with GeneTuring
Shang, X., Liao, X., Ji, Z., & Hou, W. (2025) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf492 · Cited by 53 · see citing works
“We also recognize the value of complementary benchmarks such as BioCoder [12], which focus on LLMs’ ability to perform coding tasks in bioinformatics.”
Lin, A.-Q., Ye, J., Qi, C., Zhu, L., Mou, W.-M., Gan, W.-Y., Zeng, D.-Q., Tang, B.-F., Xiao, M.-J., Chu, G., et al. (2025) · Briefings Bioinform. · doi:10.1093/bib/bbaf357 · Cited by 43 · see citing works
“Therefore, developing standardized evaluation benchmarks (such as BioCoder [120] developed by Tang et al .) will facilitate objective comparisons between different methods and help ensure the credibility and reproducibility of research results.”
Can Large Language Models Write Parallel Code?
Nichols, D., Davis, J. H., Xie, Z., Rajaram, A., & Bhatele, A. S. (2024) · ACM International Symposium on High-Performance Parallel and Distributed Computing (HPDC) · doi:10.1145/3625549.3658689 · Cited by 36 · see citing works
Bioinformatics and biomedical informatics with ChatGPT: Year one review
Wang, J., Cheng, Z., Yao, Q., Liu, L., Xu, D. H., & Hu, G. (2024) · Quantitative Biology · doi:10.1002/qub2.67 · Cited by 34 · see citing works
Wang, J., Ye, Q., Liu, L., Guo, N. L., & Hu, G. (2024) · npj Precision Oncology · doi:10.1038/s41698-024-00576-z · Cited by 25 · see citing works
Transformers in source code generation: A comprehensive survey
Ghaemi, H., Alizadehsani, Z., Shahraki, A., & Corchado, J. M. (2024) · Journal of Systems Architecture · doi:10.1016/j.sysarc.2024.103193 · Cited by 23 · see citing works
BioAgents: Bridging the gap in bioinformatics analysis with multi-agent systems
Mehandru, N., Hall, A. K., Melnichenko, O., Dubinina, Y., Tsirulnikov, D., Bamman, D., Alaa, A., Saponas, S., & Malladi, V. S. (2025) · Scientific Reports · doi:10.1038/s41598-025-25919-z · Cited by 19 · see citing works
OpenHands: An Open Platform for AI Software Developers as Generalist Agents
Wang, X., Li, B., Song, Y., Xu, F. F., Tang, X., Zhuge, M., Pan, J., Song, Y., Li, B., Singh, J., et al. (2024) · arXiv · doi:10.48550/arxiv.2407.16741 · Cited by 24 · see citing works
“BioCoder (Tang et al., 2024c) is a repository-level code generation benchmark that evaluates agents’ performance on bioinformatics-related tasks, specifically the ability to retrieve and accurately utilize context.”
Self-citationCinquin, O. (2024) · Briefings in Bioinformatics · doi:10.1093/bib/bbaf045 · Cited by 10 · see citing works
Making large language models reliable data science programming copilots for biomedical research
Wang, Z., Danek, B. P., Yang, Z., Chen, Z., & Sun, J. (2026) · Nature Biomedical Engineering · doi:10.1038/s41551-025-01587-2 · Cited by 11 · see citing works
“In medicine, recent work has introduced LLMs to automate machine learning modelling 18 , support bioinformatic tool development 19 and validate hypotheses 20,21 .”
Benchmarking LLM-based agents for single-cell omics analysis
Liu, Y., Zhou, L., Du, X., He, R., Zhang, X., Shen, R., & Li, Y. (2026) · Genome biology · doi:10.1186/s13059-026-03998-z · Cited by 11 · see citing works
“Since existing benchmarks 16,18,19 primarily focus on quantitative metrics like task success rate and code success rate, and we also believe that task completion quality (pass rate, success rate, consistency, etc.) best reflects an agent’s capability, we assign it the highest weight.”
Chen, Z., Chen, S., Ning, Y., Zhang, Q., Wang, B., Yu, B., Li, Y., Liao, Z., Wei, C., Lu, Z., et al. (2024) · arXiv · doi:10.48550/arxiv.2410.05080 · Cited by 7 · see citing works
Zheng, D., Wang, Y., Shi, E., Zhang, R., Ma, Y., Zhang, H., & Zheng, Z. (2025) · 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE) · doi:10.1109/icse55347.2025.00228 · Cited by 11 · see citing works
Koohestani, R., de Bekker, P., Koç, B., & Izadi, M. (2025) · IEEE Transactions on Software Engineering · doi:10.1109/tse.2025.3644183 · Cited by 5 · see citing works
AI-Empowered Genome Decoding: Applications of Large Language Models in Genomics
Li, S., Fan, W., & Zhou, Y. (2025) · Frontiers of digital education. · doi:10.1007/s44366-025-0051-1 · Cited by 5 · see citing works
Bioinformatics Illustrations Decoded by ChatGPT: The Good, The Bad, and The UglyPreprint
Wang, J., Ye, Q., Liu, L., Guo, N. L., & Hu, G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.15.562423 · Cited by 4 · see citing works
In-silico evaluation of aging-related interventions using omics data and predictive modeling
Fuellen, G., Palmer, D. H., Fruijtier-Pölloth, C., & Avelar, R. A. (2025) · Ageing Research Reviews · doi:10.1016/j.arr.2025.102777 · Cited by 2 · see citing works
Review of Large Language Model-Based Software Fault Localization Techniques
Tian, P., Liu, P., & Li, Y. (2025) · International Conference on Dependable Systems and Their Applications (DSA) · doi:10.1109/dsa66321.2025.00021 · Cited by 2 · see citing works
“Advanced LLMs such as GPT-4 [3,4], Claude [5], Llama-3 [6], DeepSeek [5], and domain-specialized models like CodeT5 [7] or StarCoder [8] have demonstrated promising performance in code comprehension, natural language reasoning, and multi-step problem solving.”
Ang, M. Y., Chen, L., Song, L., Lipovich, L., & Choo, S. W. (2026) · Life · doi:10.3390/life16061032 · Cited by 2 · see citing works
“LLMs can generate R, Python, Bash, Perl, Snakemake, or Nextflow scripts; explain error messages; suggest workflow structures; draft README files; and help users understand command-line tools [57,58].”
Rodrigues, W. F., Parise, M. T. D., Parise, D., dos Santos, L. M., Ribeiro, P. D. S., Ristow, P., Cardoso, M. S., Azevedo, V., Soares, S. D. C., de Melo-Minardi, R. C., et al. (2026) · Frontiers in Bioinformatics · doi:10.3389/fbinf.2026.1829576 · Cited by 2 · see citing works
“…curated databases, illustrate a complementary direction in which language models are connected to external biomedical resources to support genomic question answering, literature curation, variant interpretation, and evidence-grounded reporting (Du et al., 2026; Jin et al., 2024; Tang et al., 2024).”
InfluentialAgentic genomics: From pipeline automation to autonomous validation
Corpas, M., Guio, H., & Fatumo, S. (2026) · Cell Genomics · doi:10.1016/j.xgen.2026.101305 · Cited by 2 · see citing works
CodeDiffuSe: A masked diffusion framework for structure-aware code completion and repair
Onan, A., & Alhumyani, H. A. (2025) · Journal of King Saud University - Computer and Information Sciences · doi:10.1007/s44443-025-00237-6 · Cited by 2 · see citing works
Prompt-based bioinformatic pipeline generation for a multi-step metaviral workflow
Ma, P., Zheng, H., Yi, W., Ma, L., Sigmon, B., Weber, K. A., Hu, G., & Yao, Q. (2025) · Bioinformatics Advances · doi:10.1093/bioadv/vbaf308 · Cited by 1 · see citing works
“This benchmark encompasses a wide range of tasks, such as sequence analysis, and includes a diverse set of programming challenges (Tang et al. 2024).”
RealisticCodeBench: Towards More Realistic Evaluation of Large Language Models for Code Generation
Yu, X., Chen, H., Liu, L., Hu, X., Keung, J., & Xia, X. (2025) · International Conference on Automated Software Engineering · doi:10.1109/ase63991.2025.00248 · Cited by 3 · see citing works
“The 1 This literature review was conducted in April 2025. first category, comprising 25 papers [39], [40], [41], [42], [43], [44], [45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], focuses on domain-specific code generation abilities,…”
Forero, D. A. (2026) · Current Genomics · doi:10.2174/0113892029403403260112075225 · Cited by 1 · see citing works
“Of particular relevance, code created by LLMs might be useful to lower the entry barrier for the development of advanced computational methods and prototypes [45, 47-49] -a concept recently called vibe coding [50]-, in addition to serving as a teaching tool for wet lab scientists [40, 51-55].”
Repository-Level Code Generation Method Enhanced by Context-Dependent Graph Retrieval
Zhang, H., Li, A., Kuai, Z., Wei, Z., Kuang, L., & Xia, Y. (2025) · Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications Engineering · doi:10.1007/978-3-031-93257-1_3
Out-of-the-box bioinformatics capabilities of large language models (LLMs)
Rajesh, V., & Siwo, G. H. (2025) · bioRxiv · doi:10.1101/2025.08.22.671610 · Cited by 1 · see citing works
“…consists of 2400 multi-choice questions on practical biology tasks with a small subset on DNA and protein sequences [15], BaisBench performs assessment of LLMs on single-cell analysis tasks [16] and BioCoder which is a benchmark dataset for evaluating LLMs in bioinformatics-code generation [17].”
Alvi, R., Zaman, S. B., Karim, W., Abian, A. I., Raiaan, M. A. K., Mukta, S., Rashid, M. R. U., Islam, M. R., Sebastian, Y., & Azam, S. (2025) · Archives of Computational Methods in Engineering · doi:10.1007/s11831-026-10743-z · Cited by 3 · see citing works
“For example, General-purpose models such as GPT-4 show strong capabilities in workflow management and code generation [58, 59]; however, they are consistently outperformed by domain-specialized models on specific predictive tasks.”
Skill-Augmented Frontier Agents Nearly Saturate BixBench-Verified-50Preprint
Zhang, X. (2026) · bioRxiv · doi:10.64898/2026.04.28.721523
Hou, Y.-F., Wang, J.-J., Xiang, K., Long, Q.-Q., Zhou, Y.-C., & Meng, Z. (2026) · Knowledge Discovery and Data Mining · doi:10.1145/3770855.3817576 · Cited by 1 · see citing works
Yan, J., Fang, X., Zhu, J., Wu, C., Yang, Y., Du, C., Fang, M., Zhang, K., Zhang, Z., & Liu, Q. (2026) · ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) · doi:10.1145/3770855.3817556
Zheng, D., Wang, Y., Shi, E., Liu, X., Ma, Y., Zhang, H., & Zheng, Z. (2026) · ACM/IEEE International Conference on Software Engineering: Companion Proceedings (ICSE-Companion) · doi:10.1145/3744916.3773271 · Cited by 12 · see citing works
- Authors
- Tang, X., Qian, B., Gao, R., Chen, J., Chen, X., & Gerstein, M. B.
- Affiliations
- Yale University (US)
- Google (United States) (US)
- Corresponding author
- Gerstein, Mark B
- Venue
- Bioinformatics
- Publisher
- Oxford University Press (OUP)
- Published
- 2024-06-28
- Volume
- 40
- Issue
- Supplement_1
- Pages
- i266-i276
- ISSN
- 1367-4803 (ISSN-L)
- 1367-4811
- PMC
- PMC11211839
- OpenAlex
- W4400140877
- Cited by
- 30
- References
- 6
- Citations by year
- 2026: 11 · 2025: 13 · 2024: 5 · 2023: 1
- FWCI
- 10.90
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Licence
- Locations
- Bioinformatics (publishedVersion, cc-by)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- arXiv (Cornell University) (submittedVersion, cc-by)
- arXiv (Cornell University) (cc-by)
- Funders
- Schmidt Futures
- Topics
- Physical Sciences › Computer Science › Information Systems › Software Engineering Research
- Physical Sciences › Computer Science › Artificial Intelligence › Topic Modeling
- Social Sciences › Decision Sciences › Information Systems and Management › Scientific Computing and Data Management
- Keywords
- large language models, benchmark, Python, Java, global variables, fuzzing, StarCoder, GPT-4, prompt length, domain knowledge, fine-tuning
- MeSH terms
- Algorithms, Programming Languages, Software, Computational Biology, Benchmarking
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
