CoChat
Sign Up
PreprintCited by 4Machine Learning in BioinformaticsArtificial Intelligence in Healthcare and EducationArtificial Intelligence Applications

An Evaluation of Large Language Models in Bioinformatics Research

Open access
Yin, Hengchuang, Gu, Zhonghui, Wang, Fanhao, Abuduhaibaier, Yiparemu, Zhu, Yanqiao, Tu, Xinming, Hua, Xian-Sheng, Luo, Xiao +1 more·2024·arXiv
DOI 10.48550/arxiv.2402.13714arXiv 2402.13714
Open this paper in CoChat — free accountOpen PDF

Abstract

Large language models (LLMs) such as ChatGPT have gained considerable interest across diverse research communities. Their notable ability for text completion and generation has inaugurated a novel paradigm for language-interfaced problem solving. However, the potential and efficacy of these models in bioinformatics remain incompletely explored. In this work, we study the performance LLMs on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks. In addition, we provide a thorough analysis of their limitations in the context of complicated bioinformatics tasks. In conclusion, we believe that this work can provide new perspectives and motivate future research in the field of LLMs applications, AI for Science and bioinformatics.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

DataCite · 10.48550/arxiv.2402.13714

Cites

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Claims other papers make about this one

Each is checked against this paper’s own text: does it say what the citing paper says it does?

  • AI-assisted data analysis has emerged as one of the most powerful and democratizing applications of LLMs in scientific research.

    “AI-assisted data analysis has emerged as one of the most powerful and democratizing applications of LLMs in scientific research 19 .”

    From Reimagining biomedical science workflows in the age of large language models (2026) · cites this paper as [19]

    The Introduction (page not provided) reports that LLMs can partially solve several bioinformatics tasks and can match competitive baselines on simple tasks, supporting their potential for AI-assisted analysis. It does not establish that data analysis is among the most powerful applications of LLMs across scientific research or that it democratizes research. (LLM, conf=0.85)

    Experimental — accuracy not yet measured

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

No references could be resolved yet.

Research works printed but not linked: 85

  1. A. Birhane, A. Kasirzadeh, D. Leslie, and S. Wachter, "Science in the age of large language models," Nature Reviews Physics, pp. 1-4, 2023.
  2. U. Katz, M. Geva, and J. Berant, "Inferring implicit relations in complex questions with language models," in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 2548-2566.
  3. X. L. Li, A. Kuncoro, J. Hoffmann, C. de Masson d'Autume, P. Blun- som, and A. Nematzadeh, "A systematic investigation of commonsense knowledge in large language models," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11 838-11 855.
  4. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., "Training language models to follow instructions with human feedback," in Proceedings of the Conference on Neural Information Processing Systems, 2022, pp. 27 730-27 744.
  5. F. Xu, Q. Lin, J. Han, T. Zhao, J. Liu, and E. Cambria, "Are large language models really good logical reasoners? a comprehensive eval- uation from deductive, inductive and abductive views," arXiv preprint arXiv:2306.09841, 2023. · arXiv
  6. B. Pang, L. Lee, and S. Vaithyanathan, "Thumbs up? sentiment classi- fication using machine learning techniques," arXiv preprint cs/0205070, 2002. · arXiv
  7. M. Marrero, J. Urbano, S. Sánchez-Cuadrado, J. Morato, and J. M. Gómez-Berbís, "Named entity recognition: fallacies, challenges and opportunities," Computer Standards & Interfaces, vol. 35, no. 5, pp. 482-489, 2013.
  8. K. Lee, D. Palsetia, R. Narayanan, M. M. A. Patwary, A. Agrawal, and A. Choudhary, "Twitter trending topic classification," in Proceedings of the International Conference on Data Mining Workshops, 2011, pp. 251-258.
  9. Y. Liu, T. Han, S. Ma, J. Zhang, Y. Yang, J. Tian, H. He, A. Li, M. He, Z. Liu et al., "Summary of chatgpt/gpt-4 research and per- spective towards the future of large language models," arXiv preprint arXiv:2304.01852, 2023. · arXiv
  10. T. Sun, Z. He, H. Qian, Y. Zhou, X.-J. Huang, and X. Qiu, "Bbtv2: towards a gradient-free future with large language models," in Pro- ceedings of the Conference on Empirical Methods in Natural Language Processing, 2022, pp. 3916-3930.
  11. T. Zhang, F. Ladhak, E. Durmus, P. Liang, K. McKeown, and T. B. Hashimoto, "Benchmarking large language models for news summa- rization," arXiv preprint arXiv:2301.13848, 2023. · arXiv
  12. I. Beltagy, A. Cohan, R. Logan IV, S. Min, and S. Singh, "Zero- and few-shot nlp with pretrained language models," in Proceedings of the Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, 2022, pp. 32-37.
  13. Y. Bang, S. Cahyawijaya, N. Lee, W. Dai, D. Su, B. Wilie, H. Lovenia, Z. Ji, T. Yu, W. Chung et al., "A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity," arXiv preprint arXiv:2302.04023, 2023. · arXiv
  14. W. Jiao, W. Wang, J.-t. Huang, X. Wang, and Z. Tu, "Is chatgpt a good translator? a preliminary study," arXiv preprint arXiv:2301.08745, 2023. · arXiv
  15. Y. Tan, D. Min, Y. Li, W. Li, N. Hu, Y. Chen, and G. Qi, "Evaluation of chatgpt as a question answering system for answering complex questions," arXiv preprint arXiv:2303.07992, 2023. · arXiv
  16. T. H. Kung, M. Cheatham, A. Medenilla, C. Sillos, L. De Leon, C. Elepaño, M. Madriaga, R. Aggabao, G. Diaz-Candido, J. Maningo et al., "Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models," PLoS Digital Health, vol. 2, no. 2, p. e0000198, 2023.
  17. S. B. Patel and K. Lam, "Chatgpt: the future of discharge summaries?" The Lancet Digital Health, vol. 5, no. 3, pp. e107-e108, 2023.
  18. Q. Lu, D. Dou, and T. Nguyen, "Clinicalt5: A generative language model for clinical text," in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 5436-5443.
  19. J. Otmakhova, K. Verspoor, T. Baldwin, A. J. Yepes, and J. H. Lau, "M3: Multi-level dataset for multi-document summarisation of medical studies," in Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 3887-3901.
  20. Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R. Verkuil, O. Kabeli, Y. Shmueli et al., "Evolutionary-scale prediction of atomic- level protein structure with a language model," Science, vol. 379, no. 6637, pp. 1123-1130, 2023.
  21. A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y. Wang, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger et al., "Prottrans: Toward understanding the language of life through self-supervised learning," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 10, pp. 7112-7127, 2021.
  22. H. Lee, S. Lee, I. Lee, and H. Nam, "Amp-bert: Prediction of antimicro- bial peptide function based on a bert model," Protein Science, vol. 32, no. 1, p. e4529, 2023.
  23. I. Jahan, M. T. R. Laskar, C. Peng, and J. Huang, "Evaluation of chatgpt on biomedical tasks: A zero-shot comparison with fine-tuned generative transformers," arXiv preprint arXiv:2306.04504, 2023. · arXiv
  24. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale et al., "Llama 2: Open foundation and fine-tuned chat models," arXiv preprint arXiv:2307.09288, 2023. · arXiv
  25. Ö. AYDIN, "Google bard generated literature review: metaverse," Jour- nal of AI, vol. 7, no. 1, pp. 1-14, 2023.
  26. M. Wysocka, O. Wysocki, M. Delmas, V. Mutel, and A. Freitas, "Large language models, scientific knowledge and factuality: A systematic analysis in antibiotic discovery," arXiv preprint arXiv:2305.17819, 2023. · arXiv
  27. Y. Liu, S. Feng, D. Wang, Y. Zhang, and H. Schütze, "Evaluate what you can't evaluate: Unassessable generated responses quality," arXiv preprint arXiv:2305.14658, 2023. · arXiv
  28. S. S. Biswas, "Role of chat gpt in public health," Annals of Biomedical Engineering, vol. 51, no. 5, pp. 868-869, 2023.
  29. C. Shyr, Y. Hu, P. A. Harris, and H. Xu, "Identifying and extracting rare disease phenotypes with large language models," arXiv preprint arXiv:2306.12656, 2023. · arXiv
  30. X. Li, Y. Zhang, and E. C. Malthouse, "A preliminary study of chatgpt on news recommendation: Personalization, provider fairness, fake news," arXiv preprint arXiv:2306.10702, 2023. · arXiv
  31. D. Pu and V. Demberg, "Chatgpt vs human-authored text: Insights into controllable text summarization and sentence style transfer," arXiv preprint arXiv:2306.07799, 2023. · arXiv
  32. P. Cramer, "Alphafold2 and the future of structural biology," Nature Structural & Molecular Biology, vol. 28, no. 9, pp. 704-705, 2021.
  33. J. Deng, Z. Yang, I. Ojima, D. Samaras, and F. Wang, "Artificial intelligence in drug discovery: applications and techniques," Briefings in Bioinformatics, vol. 23, no. 1, 2022.
  34. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv:1810.04805, 2018. · arXiv
  35. B. Fabian, T. Edlich, H. Gaspar, M. Segler, J. Meyers, M. Fiscato, and M. Ahmed, "Molecular representation learning with language models and domain-relevant auxiliary tasks," arXiv preprint arXiv:2011.13230, 2020. · arXiv
  36. E. Shue, L. Liu, B. Li, Z. Feng, X. Li, and G. Hu, "Empowering beginners in bioinformatics with chatgpt," bioRxiv, pp. 2023-03, 2023.
  37. S. R. Piccolo, P. Denny, A. Luxton-Reilly, S. Payne, and P. G. Ridge, "Many bioinformatics programming tasks can be automated with chat- gpt," arXiv preprint arXiv:2303.13528, 2023. · arXiv
  38. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu, "Biogpt: generative pre-trained transformer for biomedical text generation and mining," Briefings in Bioinformatics, vol. 23, no. 6, Sep. 2022. [Online]. Available: http://dx.doi.org/10.1093/bib/bbac409 · DOI · Paper page
  39. D. B. Lubahn, T. R. Brown, J. A. Simental, H. N. Higgs, C. J. Migeon, E. M. Wilson, and F. S. French, "Sequence of the intron/exon junctions of the coding region of the human androgen receptor gene and identification of a point mutation in a family with complete androgen insensitivity." Proceedings of the National Academy of Sciences, vol. 86, no. 23, pp. 9534-9538, 1989.
  40. W. Zhu, A. Lomsadze, and M. Borodovsky, "Ab initio gene identification in metagenomic sequences ," Nucleic Acids Research, vol. 38, no. 12, pp. e132-e132, 2010.
  41. J. H. Badger and G. J. Olsen, "Critica: coding region identification tool invoking comparative analysis." Molecular Biology and Evolution, vol. 16, no. 4, pp. 512-524, 1999.
  42. R. Wise, T. Hart, O. Cars, M. Streulens, R. Helmuth, P. Huovinen, and M. Sprenger, "Antimicrobial resistance," pp. 609-610, 1998.
  43. A. A. Bahar and D. Ren, "Antimicrobial peptides," Pharmaceuticals, vol. 6, no. 12, pp. 1543-1575, 2013.
  44. M. Zasloff, "Antimicrobial peptides of multicellular organisms," Nature, vol. 415, no. 6870, pp. 389-395, 2002.
  45. B. Liu, L. Ezeogu, L. Zellmer, B. Yu, N. Xu, and D. Joshua Liao, "Protecting the normal in order to better kill the cancer," Cancer Medicine, vol. 4, no. 9, pp. 1394-1403, 2015.
  46. J. Li, S. Tan, X. Chen, C.-Y. Zhang, and Y. Zhang, "Peptide aptamers with biological and therapeutic applications," Current Medicinal Chem- istry, vol. 18, no. 27, pp. 4215-4222, 2011.
  47. A. Tyagi, A. Tuknait, P. Anand, S. Gupta, M. Sharma, D. Mathur, A. Joshi, S. Singh, A. Gautam, and G. P. Raghava, "Cancerppd: a database of anticancer peptides and proteins," Nucleic Acids Research, vol. 43, no. D1, pp. D837-D843, 2015.
  48. W. Chiangjong, S. Chutipongtanate, and S. Hongeng, "Anticancer pep- tide: Physicochemical property, functional aspect and trend in clinical application," International Journal of Oncology, vol. 57, no. 3, pp. 678- 696, 2020.
  49. Q. Li, W. Zhou, D. Wang, S. Wang, and Q. Li, "Prediction of anti- cancer peptides using a low-dimensional feature model," Frontiers in Bioengineering and Biotechnology, vol. 8, p. 892, 2020.
  50. M. L. Verdonk and M. J. Hartshorn, "Structure-guided fragment screen- ing for lead discovery." Current Opinion in Drug Discovery & Devel- opment, vol. 7, no. 4, pp. 404-410, 2004.
  51. J. M. Sangster, Octanol-water partition coefficients: fundamentals and physical chemistry. John Wiley & Sons, 1997, vol. 1.
  52. B. Ouyang, J. Wang, T. He, C. J. Bartel, H. Huo, Y. Wang, V. Lacivita, H. Kim, and G. Ceder, "Synthetic accessibility and stability rules of nasicons," Nature Communications, vol. 12, no. 1, p. 5752, 2021.
  53. G. R. Bickerton, G. V. Paolini, J. Besnard, S. Muresan, and A. L. Hopkins, "Quantifying the chemical beauty of drugs," Nature Chemistry, vol. 4, no. 2, pp. 90-98, 2012.
  54. E. A. Bruford, B. Braschi, P. Denny, T. E. Jones, R. L. Seal, and S. Tweedie, "Guidelines for human gene nomenclature," Nature Ge- netics, vol. 52, no. 8, pp. 754-758, 2020.
  55. K. Fundel and R. Zimmer, "Gene and protein nomenclature in public databases," BMC Bioinformatics, vol. 7, no. 1, pp. 1-13, 2006.
  56. T. C. Rindflesch, L. Tanabe, J. N. Weinstein, and L. Hunter, "Edgar: extraction of drugs, genes and relations from the biomedical literature," in Biocomputing 2000. World Scientific, 1999, pp. 517-528.
  57. C. Blaschke, L. Hirschman, and A. Valencia, "Information extraction in molecular biology," Briefings in Bioinformatics, vol. 3, no. 2, pp. 154-165, 2002.
  58. A. Yeh, A. Morgan, M. Colosimo, and L. Hirschman, "Biocreative task 1a: gene mention finding evaluation," BMC Bioinformatics, vol. 6, pp. 1-10, 2005.
  59. M. F. Wangler, S. Yamamoto, and H. J. Bellen, "Fruit flies in biomedical research," Genetics, vol. 199, no. 3, pp. 639-653, 2015.
  60. S. J. Goebel, G. P. Johnson, M. E. Perkus, S. W. Davis, J. P. Winslow, and E. Paoletti, "The complete dna sequence of vaccinia virus," Virology, vol. 179, no. 1, pp. 247-266, 1990.
  61. Z. Chen, M. R. Min, S. Parthasarathy, and X. Ning, "A deep generative model for molecule optimization via one fragment modification," Nature Machine Intelligence, vol. 3, no. 12, pp. 1040-1049, 2021.
  62. T. Greenhalgh, "How to read a paper: the medline database," Bmj, vol. 315, no. 7101, pp. 180-183, 1997.
  63. M. Furuno, T. Kasukawa, R. Saito, J. Adachi, H. Suzuki, R. Baldarelli, Y. Hayashizaki, and Y. Okazaki, "Cds annotation in full-length cdna sequence," Genome Research, vol. 13, no. 6b, pp. 1478-1487, 2003.
  64. T. Chen, T. He, M. Benesty, V. Khotilovich, Y. Tang, H. Cho, K. Chen, R. Mitchell, I. Cano, T. Zhou et al., "Xgboost: extreme gradient boosting," R package version 0.4-2, vol. 1, no. 4, pp. 1-4, 2015.
  65. A. M. Kibriya, E. Frank, B. Pfahringer, and G. Holmes, "Multinomial naive bayes for text categorization revisited," in AI 2004: Advances in Artificial Intelligence: 17th Australian Joint Conference on Artificial Intelligence, Cairns, Australia, December 4-6, 2004. Proceedings 17, 2005, pp. 488-499.
  66. V. Jakkula, "Tutorial on support vector machine (svm)," School of EECS, Washington State University, vol. 37, no. 2.5, p. 3, 2006.
  67. G. Guo, H. Wang, D. Bell, Y. Bi, and K. Greer, "Knn model-based ap- proach in classification," in On The Move to Meaningful Internet Systems 2003: CoopIS, DOA, and ODBASE: OTM Confederated International Conferences, CoopIS, DOA, and ODBASE 2003, Catania, Sicily, Italy, November 3-7, 2003. Proceedings, 2003, pp. 986-996.
  68. M. Maalouf, "Logistic regression in data analysis: an overview," Inter- national Journal of Data Analysis Techniques and Strategies, vol. 3, no. 3, pp. 281-299, 2011.
  69. A. Pinkus, "Approximation theory of the mlp model in neural networks," Acta Numerica, vol. 8, pp. 143-195, 1999.
  70. G. Biau and E. Scornet, "A random forest guided tour," TEST, vol. 25, pp. 197-227, 2016.
  71. H. Saigo, S. Nowozin, T. Kadowaki, T. Kudo, and K. Tsuda, "gboost: a mathematical programming approach to graph classification and regres- sion," Machine Learning, vol. 75, pp. 69-89, 2009.
  72. S. A. Wildman and G. M. Crippen, "Prediction of physicochemical parameters by atomic contributions," Journal of Chemical Information and Computer Sciences, vol. 39, no. 5, pp. 868-873, 1999.
  73. P. Ertl and A. Schuffenhauer, "Estimation of synthetic accessibility score of drug-like molecules based on molecular complexity and fragment contributions," Journal of Cheminformatics, vol. 1, pp. 1-11, 2009.
  74. W. Jin, K. Yang, R. Barzilay, and T. Jaakkola, "Learning multimodal graph-to-graph translation for molecular optimization," arXiv preprint arXiv:1812.01070, 2018. · arXiv
  75. T. Sterling and J. J. Irwin, "Zinc 15-ligand discovery for everyone," Journal of Chemical Information and Modeling, vol. 55, no. 11, pp. 2324-2337, 2015.
  76. H. Cho and H. Lee, "Biomedical named entity recognition using deep neural networks with contextual information," BMC bioinformatics, vol. 20, pp. 1-11, 2019.
  77. H. Zhao, Y. Yang, Q. Zhang, and L. Si, "Improve neural entity recognition via multi-task data selection and constrained decoding," in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 2018, pp. 346-351.
  78. J. Lee, W. Yoon, S. Kim, D. Kim, S. Kim, C. H. So, and J. Kang, "Biobert: a pre-trained biomedical language representation model for biomedical text mining," Bioinformatics, vol. 36, no. 4, pp. 1234-1240, 2020.
  79. T. Zhang, C. Xia, P. S. Yu, Z. Liu, and S. Zhao, "Pdaln: Progressive domain adaptation over a pre-trained model for low-resource cross- domain named entity recognition," in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2021.
  80. R. A. Baeza-Yates, "Algorithms for string searching," in ACM SIGIR Forum, vol. 23, no. 3-4, 1989, pp. 34-58.
  81. L. Lovász and H. J. Prömel, "Combinatorics," Oberwolfach Reports, vol. 1, no. 1, pp. 5-110, 2004.
  82. R. E. Bellman and S. E. Dreyfus, Applied dynamic programming. Princeton University Press, 2015, vol. 2050.
  83. R. C. Edgar and S. Batzoglou, "Multiple sequence alignment," Current Opinion in Structural Biology, vol. 16, no. 3, pp. 368-373, 2006.
  84. S. Even, Graph algorithms. Cambridge University Press, 2011.
  85. K. G. Field, G. J. Olsen, D. J. Lane, S. J. Giovannoni, M. T. Ghiselin, E. C. Raff, N. R. Pace, and R. A. Raff, "Molecular phylogeny of the animal kingdom," Science, vol. 239, no. 4841, pp. 748-753, 1988.

Cited by

Sorted by citations

Authors
Yin, H., Gu, Z., Wang, F., Abuduhaibaier, Y., Zhu, Y., Tu, X., Hua, X.-S., Luo, X., & Sun, Y.
Affiliations
  • Peking University (CN)
  • Henan University (CN)
ORCID
Venue
arXiv
Publisher
arXiv
Published
2024
ISSN
2331-8422 (ISSN-L)
DOI
10.48550/arxiv.2402.13714
arXiv
2402.13714
OpenAlex
W4392090049
Cited by
4
References
0
Citations by year
2026: 1 · 2025: 1 · 2024: 2
Type
posted content
Language
en
Open access
Licence
Locations
  • arXiv (Cornell University) (submittedVersion, cc-by)
  • arXiv (Cornell University) (cc-by)
Topics
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
  • Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
  • Physical Sciences › Computer Science › Artificial Intelligence › Artificial Intelligence Applications
Keywords
large language models, ChatGPT, named entity recognition, antimicrobial peptides, anticancer peptides, molecular optimization, prompt engineering, artificial intelligence in science

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review