The DOI matches its Crossref record; 13 of the 13 references it cites were found in a registry or an index.
Analysis of large-language model versus human performance for genetics questions
Open accessPublished version of a preprint
Abstract
Large-language models like ChatGPT have recently received a great deal of attention. One area of interest pertains to how these models could be used in biomedical contexts, including related to human genetics. To assess one facet of this, we compared the performance of ChatGPT versus human respondents (13,642 human responses) in answering 85 multiple-choice questions about aspects of human genetics. Overall, ChatGPT did not perform significantly differently (p = 0.8327) than human respondents; ChatGPT was 68.2% accurate, compared to 66.6% accuracy for human respondents. Both ChatGPT and humans performed better on memorization-type questions versus critical thinking questions (p < 0.0001). When asked the same question multiple times, ChatGPT frequently provided different answers (16% of initial responses), including for both initially correct and incorrect answers, and gave plausible explanations for both correct and incorrect answers. ChatGPT's performance was impressive, but currently demonstrates significant shortcomings for clinical or other high-stakes use. Addressing these limitations will be important to guide adoption in real-life situations.
Overview
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
In the citing paper
“It should also be noted that a replication of this study might not yield the exact same results, as the literature suggests that GPT-3.5 is inconsistent in answering 15% of medical questions [59].”
From Comparison of the Performance of GPT-3.5 and GPT-4 With That of Medical Students on the Written German Medical Licensing Examination: Observational Study (2024) · cites this paper as [59]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Large Language Models Encode Clinical Knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2022) · arXiv · doi:10.48550/arxiv.2212.13138 · Cited by 4,083
Large language models encode clinical knowledge
Singhal, K., Azizi, S., Tu, T., Mahdavi, S. S., Wei, J., Chung, H. W., Scales, N., Tanwani, A., Cole-Lewis, H., Pfohl, S., et al. (2023) · Nature · doi:10.1038/s41586-023-06291-2 · Cited by 4,083
Predicting splicing from primary sequence with deep learning
Jaganathan, K., Kyriazopoulou Panagiotopoulou, S., McRae, J. F., Darbandi, S. F., Knowles, D., Li, Y. I., Kosmicki, J. A., Arbelaez, J., Cui, W., Schwartz, G. B., et al. (2019) · Cell · doi:10.1016/j.cell.2018.12.015 · Cited by 2,853
A universal SNP and small-indel variant caller using deep neural networks
Poplin, R., Chang, P.-C., Alexander, D., Schwartz, S., Colthurst, T., Ku, A., Newburger, D., Dijamco, J., Nguyen, N., Afshar, P. T., et al. (2018) · Nature Biotechnology · doi:10.1038/nbt.4235 · Cited by 1,840
A large language model for electronic health records
Yang, X., Chen, A., PourNejatian, N., Shin, H. C., Smith, K. E., Parisien, C., Compas, C., Martin, C., Costa, A. B., Flores, M. G., et al. (2022) · npj Digital Medicine · doi:10.1038/s41746-022-00742-2 · Cited by 847
AI for radiographic COVID-19 detection selects shortcuts over signal
DeGrave, A. J., Janizek, J. D., & Lee, S.-I. (2021) · Nature Machine Intelligence · doi:10.1038/s42256-021-00338-7 · Cited by 631
Artificial intelligence in clinical and genomic diagnostics
Dias, R., & Torkamani, A. (2019) · Genome Medicine · doi:10.1186/s13073-019-0689-8 · Cited by 499
The use of machine learning in rare diseases: a scoping review
Schaefer, J., Lehne, M., Schepers, J., Prasser, F., & Thun, S. (2020) · Orphanet Journal of Rare Diseases · doi:10.1186/s13023-020-01424-6 · Cited by 213
Shelmerdine, S. C., Martin, H. M., Shirodkar, K. K., Shamshuddin, S., & Weir-McCall, J. R. (2022) · BMJ · doi:10.1136/bmj-2022-072826 · Cited by 45
Rubinstein–Taybi syndrome in diverse populations
Tekendo‐Ngongang, C., Owosela, B., Fleischer, N., Addissie, Y. A., Malonga, B., Badoe, E. V., Gupta, N., Moresco, A., Huckstadt, V., Ashaat, E. A., et al. (2020) · American Journal of Medical Genetics Part A · doi:10.1002/ajmg.a.61888 · Cited by 32
Scoping review and classification of deep learning in medical genetics
Hanchard, S. E. L., Dwyer, M. C., Liu, S., Hu, P., Tekendo‐Ngongang, C., Waikel, R. L., Duong, D., & Solomon, B. D. (2022) · Genetics in Medicine · doi:10.1016/j.gim.2022.04.025 · Cited by 20
Self-citationSolomon, B. D. (2022) · Wiley eBooks · doi:10.1002/9781119847212 · Cited by 7
Self-citation
Cited by
Sorted by citations
Sallam, M. (2023) · Healthcare · doi:10.3390/healthcare11060887 · Cited by 3,126
A Survey on Evaluation of Large Language Models
Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. (2024) · ACM Transactions on Intelligent Systems and Technology · doi:10.1145/3641289 · Cited by 4,254
“[19] / Duong and Solomon [36] / Hamidi and Roberts [58] / Holmes et al.”
BackgroundToward expert-level medical question answering with large language models
Singhal, K., Tu, T., Gottweis, J., Sayres, R., Wulczyn, E., Amin, M., Hou, L., Clark, K., Pfohl, S. R., Cole-Lewis, H., et al. (2025) · Nature Medicine · doi:10.1038/s41591-024-03423-7 · Cited by 1,734
A Survey on Large Language Models: Applications, Challenges, Limitations, and Practical UsagePreprint
Hadi, M. U., Al-Tashi, Q., Qureshi, R., Shah, A., Muneer, A., Irfan, M., Zafar, A., Shaikh, M. B., Akhtar, N., Hassan, S. Z., et al. (2023) · doi:10.36227/techrxiv.23589741.v1 · Cited by 475
ChatGPT in healthcare: A taxonomy and systematic review
Li, J., Dada, A., Puladi, B., Kleesiek, J., & Egger, J. (2024) · Computer Methods and Programs in Biomedicine · doi:10.1016/j.cmpb.2024.108013 · Cited by 452
Metadata
- Authors
- Duong, D., & Solomon, B. D.
- Affiliations
- National Institutes of Health (US)
- Corresponding author
- Solomon, Benjamin D.
- ORCID
- Solomon, Benjamin D.
- Venue
- European Journal of Human Genetics
- Published
- 2023-05-29
- Volume
- 32
- Issue
- 4
- Pages
- 466-468
- ISSN
- 1018-4813 (ISSN-L)
- 1476-5438
- PMC
- PMC10999420
- OpenAlex
- W4378603221
- Cited by
- 159
- References
- 12
- Citations by year
- 2026: 15 · 2025: 42 · 2024: 59 · 2023: 42
- FWCI
- 6.29
- Type
- journal article
- Language
- en
- Open access
- green
- doi.org
- Licence
- Locations
- European Journal of Human Genetics (publishedVersion)
- medRxiv (acceptedVersion)
- PubMed Central (publishedVersion)
- Europe PMC (PubMed Central) (publishedVersion)
- PubMed (publishedVersion)
- PubMed Central (submittedVersion)
- Funders
- U.S. Department of Health & Human Services | NIH | National Human Genome Research Institute (N/A)
- Topics
- Health Sciences › Medicine › Health Informatics › Artificial Intelligence in Healthcare and Education
- Health Sciences › Medicine › Health Informatics › Biomedical Text Mining and Ontologies
- Social Sciences › Education › Artificial Intelligence in Education
- Keywords
- large language models, ChatGPT, human genetics education, multiple-choice questions
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
