The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Analysis of large-language model versus human performance for genetics questions
Open accessPublished version of a preprint
Abstract
Large-language models like ChatGPT have recently received a great deal of attention. One area of interest pertains to how these models could be used in biomedical contexts, including related to human genetics. To assess one facet of this, we compared the performance of ChatGPT versus human respondents (13,642 human responses) in answering 85 multiple-choice questions about aspects of human genetics. Overall, ChatGPT did not perform significantly differently (p = 0.8327) than human respondents; ChatGPT was 68.2% accurate, compared to 66.6% accuracy for human respondents. Both ChatGPT and humans performed better on memorization-type questions versus critical thinking questions (p < 0.0001). When asked the same question multiple times, ChatGPT frequently provided different answers (16% of initial responses), including for both initially correct and incorrect answers, and gave plausible explanations for both correct and incorrect answers. ChatGPT's performance was impressive, but currently demonstrates significant shortcomings for clinical or other high-stakes use. Addressing these limitations will be important to guide adoption in real-life situations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It should also be noted that a replication of this study might not yield the exact same results, as the literature suggests that GPT-3.5 is inconsistent in answering 15% of medical questions [59].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Predicting splicing from primary sequence with deep learning
Jaganathan, K., Kyriazopoulou Panagiotopoulou, S., McRae, J. F., Darbandi, S. F., Knowles, D., Li, Y. I., Kosmicki, J. A., Arbelaez, J., Cui, W., Schwartz, G. B., et al. (2019) · Cell · doi:10.1016/j.cell.2018.12.015 · Cited by 2,853
A universal SNP and small-indel variant caller using deep neural networks
Poplin, R., Chang, P.-C., Alexander, D., Schwartz, S., Colthurst, T., Ku, A., Newburger, D., Dijamco, J., Nguyen, N., Afshar, P. T., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4235 · Cited by 1,840
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Artificial intelligence in clinical and genomic diagnostics
Dias, R., & Torkamani, A. (2019) · Genome Medicine · doi:10.1186/s13073-019-0689-8 · Cited by 499
The use of machine learning in rare diseases: a scoping review
Schaefer, J., Lehne, M., Schepers, J., Prasser, F., & Thun, S. (2020) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-020-01424-6 · Cited by 213
Shelmerdine, S. C., Martin, H. M., Shirodkar, K. K., Shamshuddin, S., & Weir-McCall, J. R. (2022) · BMJ · doi:10.1136/bmj-2022-072826 · Cited by 45
Cited by
Sorted by citations
Items 101–160 of 160
Duong, D., & Solomon, B. D. (2023) · European Journal of Human Genetics · doi:10.1038/s41431-023-01444-3 · Cited by 5
Self-citationAssessing large language model performance related to aging in genetic conditions
Othman, A. A., Flaharty, K. A., Hanchard, S. E. L., Hu, P., Duong, D., Waikel, R. L., & Solomon, B. D. (2025) · npj Aging · doi:10.1038/s41514-025-00226-z · Cited by 4
Self-citationStudent perceptions of artificial intelligence in Doctor of Physical Therapy Education
Reoli, R., Marchese, V., Duggal, A. P., & Kaplan, K. (2025) · Physiotherapy Theory and Practice · doi:10.1080/09593985.2025.2496978 · Cited by 3
Özgür, E. G., & Bekiroğlu, N. (2025) · Heliyon · doi:10.1016/j.heliyon.2025.e41925 · Cited by 4
Mori, T., Watanabe, T., & Kosugi, S. (2025) · PLoS ONE · doi:10.1371/journal.pone.0311148 · Cited by 4
Bioinformatics Illustrations Decoded by ChatGPT: The Good, The Bad, and The UglyPreprint
Wang, J., Ye, Q., Liu, L., Guo, N. L., & Hu, G. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.10.15.562423 · Cited by 4
Kerkütlüoğlu, M., Kaya, E., & Gökmen, R. (2024) · Cureus · doi:10.7759/cureus.71472 · Cited by 3
Ali, U., Shafi, I., Ahmad, J., Caceres, A. Z., Montero, T. C. C., Rehman, H. M. R. U., & Ashraf, I. (2026) · Computational and Structural Biotechnology Journal · doi:10.34133/csbj.0023 · Cited by 3
XXV IUFRO World Congress: Forest Research and Cooperation for Sustainable
Brasileira, P. F. (2019) · Pesquisa Florestal Brasileira · doi:10.4336/2019.pfb.39e201902043 · Cited by 5
A Systematic Review of Research on ChatGPT
Guan, C., Ding, D., Gupta, P., Hung, Y.-C., & Jiang, Z. (2023) · Advances in digital crime, forensics, and cyber terrorism book series · doi:10.4018/978-1-6684-8422-7.ch007 · Cited by 3
Artificial Intelligence in the repurposing of potential herbs for filariasis therapy
Wiwanitkit, S., & Wiwanitkit, V. (2024) · Journal of Vector Borne Diseases · doi:10.4103/jvbd.jvbd_153_23 · Cited by 3
Evaluation of ChatGPT's Usefulness and Accuracy in Diagnostic Surgical Pathology.Preprint
Guastafierro, V., Corbitt, D. N., Bressan, A., Fernandes, B., Mintemur, O., Magnoli, F., Ronchi, S., Rosa, S. L., Uccella, S., & Renne, S. (2024) · medRxiv · doi:10.1101/2024.03.12.24304153 · Cited by 2
CHATGPT AS A SOCRATIC ASSISTANT: DEVELOPING STUDENTS’ REFLECTIVE AND CRITICAL THINKING
Appadoo-Ramsamy, W. (2025) · PUPIL International Journal of Teaching Education and Learning · doi:10.20319/ictel.2025.178195 · Cited by 4
Reference decisions enhance LLM performance, amplified by source disclosure
Zhang, Y., Liu, Z., Bai, S., Xu, T., & Lau, R. Y. K. (2025) · Digital Health · doi:10.1177/20552076251342078 · Cited by 2
AI-powered medicine repurposed to identify new leishmaniasis treatments
Daungsupawong, H., & Wiwanitkit, V. (2025) · Journal of Parasitic Diseases · doi:10.1007/s12639-025-01838-8 · Cited by 2
Güler, N. K., & Yıldız, B. (2025) · The Journal of Educational Research · doi:10.1080/00220671.2025.2510387 · Cited by 3
AI-Powered Automation in Healthcare: A Multi-agent Approach with LLM
de Almeida, M. P., Machado, J. M., Sousa, R., & Peixoto, H. (2025) · Communications in computer and information science · doi:10.1007/978-3-032-05925-3_11 · Cited by 2
Evaluating Large Language Models on Named Entity Recognition
Ji, B., Liu, H., Li, S., Ma, J., Yu, J., & Ng, S.-K. (2026) · IEEE Transactions on Neural Networks and Learning Systems · doi:10.1109/tnnls.2025.3636917 · Cited by 2
Darwinian evolution has become dogma; AI can rescue what is salvageable
Brown, O. R., & Hullender, D. A. (2023) · Progress in Biophysics and Molecular Biology · doi:10.1016/j.pbiomolbio.2023.12.001 · Cited by 3
Hendrik, H., Permanasari, A. E., Fauziati, S., & Kusumawardani, S. S. (2023) · International Conference on Information Technology and Digital Applications (ICITDA) · doi:10.1109/icitda60835.2023.10427395 · Cited by 2
ChatGPT’s Role in Gerontology Research
Kaufmann, C. N., Bai, C., Borgia, B., Leeuwenburgh, C., Lin, Y., Mardini, M., McElroy, T., Swanson, C. W., Wimberly, K., Zapata, R., et al. (2024) · The Journals of Gerontology Series A · doi:10.1093/gerona/glae184 · Cited by 2
AI for Oncology Drug Data Harmonization — Amazon versus OpenAI
Ronquillo, J. G., South, B. R., Wiemken, T. L., Jadhav, A., WATT, S. J., Jesus, M. D., & Habtezion, A. (2024) · NEJM AI · doi:10.1056/aics2400661 · Cited by 2
Exploring the Potential of ChatGPT for the Summarization of Patient Medical Histories: A Pilot Study
Karino, M., So, C., Jinta, T., Tanaka, M., Nakamura, T., Okafuji, K., Kitamura, A., Tomishima, Y., & Nishimura, N. (2025) · Cureus · doi:10.7759/cureus.84133 · Cited by 2
Kumaresan, A., Uden, L., & Ashraf, S. Q. (2023) · Communications in computer and information science · doi:10.1007/978-3-031-34045-1_34 · Cited by 2
Beyond the hype: a comprehensive critique of ChatGPT’s applicability in genetics
Ray, P. P., & Majumder, P. (2023) · European Journal of Human Genetics · doi:10.1038/s41431-023-01442-5 · Cited by 2
An experiment on an automated literature survey of data-driven speech enhancement methods
dos Santos, A., Pereira, J., Nogueira, R. F., Masiero, B. S., Sander-Tavallaey, S., & Zea, E. (2024) · Acta Acustica · doi:10.1051/aacus/2023067 · Cited by 1
Daungsupawong, H., & Wiwanitkit, V. (2023) · Indian Journal of Allergy Asthma and Immunology · doi:10.4103/ijaai.ijaai_44_23 · Cited by 1
Benedetti, S., Frosolini, A., Catarzi, L., Vaira, L. A., Consorti, G., Paglianiti, M., Gennaro, P., & Gabriele, G. (2025) · Journal of Maxillofacial and Oral Surgery · doi:10.1007/s12663-025-02556-7 · Cited by 1
Chebly, A., Véronèse, L., & Chevret, E. (2025) · Molecular Cytogenetics · doi:10.1186/s13039-025-00717-4 · Cited by 1
Tamayo, L. P. V., Nishiyama, T., Peng, S., Wakamiya, S., & Aramaki, E. (2026) · JMIR Medical Informatics · doi:10.2196/80416 · Cited by 1
Faraji, N., Zeinali, T., Aali, S., Mellatdoust, P., & Nejad, K. G. (2025) · Journal of Burns and Wound Research · doi:10.61882/ijbwr.1.3.35 · Cited by 1
ChatGPT® and Knowledge of Brazilian Dental Ethics and Legislation
da Nóbrega, J. B. M., Gusmão, C. L. V., Laureano, I. C. C., & Santiago, B. M. (2026) · Pesquisa Brasileira em Odontopediatria e Clínica Integrada · doi:10.1590/pboci.2026.010 · Cited by 1
Hsu, C.-W., Cochrane, M., & Bambarawana, S. (2026) · International Journal of Mental Health · doi:10.1080/00207411.2025.2611527 · Cited by 1
McLeod, G. M., Hanks, B., Ashby, H., Nolder, J., Sorenson, B., & Hanson, K. (2026) · Journal of Chemical Education · doi:10.1021/acs.jchemed.5c00979 · Cited by 1
Chen, X., Liu, J., Liu, Y., Xiang, H., Guo, X., Cai, X., & Wang, X. (2026) · Thinking Skills and Creativity · doi:10.1016/j.tsc.2026.102166 · Cited by 3
Genetic analysis of neurodegenerative diseases
Grassano, M., Schindler, A. B., Traynor, B. J., & Scholz, S. W. (2026) · Journal of Clinical Investigation · doi:10.1172/jci199840 · Cited by 1
Obstet-LLM: Large Language Model for Early Prediction of SGA-LGA Newborns
Xu, G.-X., Jiang, H., Nguyen, S. V., Wu, D., Liu, Z.-D., Sha, F., & Li, Y. (2026) · Journal of Computer Science and Technology · doi:10.1007/s11390-025-5020-0 · Cited by 1
Artificial intelligence – the next generation of sequencing?
McNeill, A. (2024) · European Journal of Human Genetics · doi:10.1038/s41431-024-01595-x · Cited by 1
Bridging Today and the Future of Humanity: AI Safety in 2024 and BeyondPreprint
Han, S. (2025) · Qeios · doi:10.32388/zhm9uq · Cited by 4
BioLLMBench: A Comprehensive Benchmarking of Large Language Models in BioinformaticsPreprint
Mangul, S., Sarwal, V., Munteanu, V., Suhodolschi, T., Ciorbă, D., Eskin, E., & Wang, W. W. (2024) · Research Square · doi:10.21203/rs.3.rs-3780193/v1
Empfehlung für Gastroenterologen
Link, A. (2024) · coloproctology · doi:10.1007/s00053-024-00775-z
Daungsupawong, H., & Wiwanitkit, V. (2024) · Mustansiriya Medical Journal · doi:10.4103/mj.mj_70_23
Introduction to the Generative Artificial Intelligence Systems
Kovač, M. (2024) · Springer eBooks · doi:10.1007/978-3-031-65514-2_4
Applying The Egra Technique in Teaching Simple Present Tense and Simple Past Tense
Rita, M., Riyanti, D., & Yuliana, Y. G. S. (2024) · Jurnal Riset dan Inovasi Pembelajaran · doi:10.51574/jrip.v4i3.1798
The role of large language models in medical genetics
Merdler-Rabinowicz, R., Omar, M., Ganesh, J., Morava, É., Nadkarni, G. N., & Klang, E. (2025) · Molecular Genetics and Metabolism · doi:10.1016/j.ymgme.2025.109098
Endure or perish with use of artificial intelligence in clinical genetics settings
Ray, P. P. (2025) · European Journal of Human Genetics · doi:10.1038/s41431-025-01852-7
Intelligent accounting question-answering robot based on a large language model and knowledge graph
Shi, S., Li, G., & Wang, Y. (2025) · Nonlinear Engineering · doi:10.1515/nleng-2024-0087
Marza, K. H. O., Jadallah, O. A., Karem, M. A. B. S., Karim, Z. F., & Khalaf, M. H. (2025) · Bright Sky Publications eBooks · doi:10.62906/bs.book.395
Reliable Annotations with Less Effort: Evaluating LLM-Human Collaboration in Search Clarifications
Tavakoli, L., & Zamani, H. (2025) · ACM SIGIR International Conference on Theory of Information Retrieval (ICTIR) · doi:10.1145/3731120.3744574 · Cited by 4
Hsu, C.-W., Robbins, A., & Cartwright, T. (2025) · Research Square · doi:10.21203/rs.3.rs-7530420/v1
Can generative AI further the classroom interactional justice?
Yang, X., Gao, Y., & Liu, X. (2025) · Journal of Computing in Higher Education · doi:10.1007/s12528-025-09473-1
Wright, B. S., Kim, L. J., & Coleman, N. R. (2023) · Annals of Pharmacy Education Safety and Public Health Advocacy · doi:10.51847/hpn7ecjdva
Artificial intelligence in genomic medicine: dispelling three myths
Solomon, B. D. (2026) · npj Genomic Medicine · doi:10.1038/s41525-026-00575-y
Self-citationArtificial intelligence-driven medicine repurposing for chronic pyelonephritis: A potential solution
Wiwanitkit, S., & Wiwanitkit, V. (2026) · Journal of Medical Society · doi:10.4103/jms.jms_8_24
Hsu, C.-W., Drummond, A., Buckingham, A., Cong, K. L., & Carson, K. (2026) · International Journal of Mental Health · doi:10.1080/00207411.2026.2705598
Öğütlü, Ö. B. G., Solomon, B. D., & Çelik, Y. S. (2026) · Frontiers in Medicine · doi:10.3389/fmed.2026.1938886
Self-citationXue, Z., Fu, K., Zhang, Y., Wu, B., & Wu, J. (2026) · Journal of Medical Internet Research · doi:10.2196/97852
Tekin, Z. Y. Ö., & Mumcu, A. K. (2026) · Odontology · doi:10.1007/s10266-026-01570-6
An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge GraphPreprint
Medagbe, J. F., Kilyungi, B. M., Isaac, A., Kabua, A., Lisso, F., Wambua, S., Williams, R. W., & Prins, P. (2026) · bioRxiv · doi:10.64898/2026.10.04.756532
Metadata
- Authors
- Duong, D., & Solomon, B. D.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Solomon, Benjamin D.
- ORCID
- Solomon, Benjamin D.
- Venue
- European Journal of Human Genetics
- Published
- 2023-05-29
- Volume
- 32
- Issue
- 4
- Pages
- 466-468
- ISSN
- 1018-4813 (ISSN-L)
- 1476-5438
- PMC
- PMC10999420
- OpenAlex
- W4378603221
- Cited by
- 159
- References
- 12
- Citations by year
- 2026: 15 · 2025: 42 · 2024: 59 · 2023: 42
- FWCI
- 6.29
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- European Journal of Human Genetics (publishedVersion)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- U.S. Department of Health & Human Services | NIH | National Human Genome Research Institute (N/A)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- large language models, ChatGPT, human genetics education, multiple-choice questions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
