PreprintCited by 123SARS-CoV-2 and COVID-19 ResearchGenomics and Phylogenetic StudiesMachine Learning in Bioinformatics

GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Open access
Zvyagin, Maxim, Brace, Alexander, Hippe, Kyle, Deng, Yuntian, Zhang, Bin, Bohorquez, Cindy Orozco, Clyde, Austin, Kale, Bharat +26 more·2022·The International Journal of High Performance Computing Applications

Preprint — published as 10.1177/10943420231201154 · Paper page

Abstract

We seek to transform how new and emergent variants of pandemiccausing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pretraining on over 110 million prokaryotic gene sequences and finetuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2022.10.10.511571

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Cites

Flagged first, then by citations

Cited by

Sorted by citations

Authors
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., Mann, C. M., Irvin, M., Gregory Pauloski, J., Ward, L., Hayot-Sasson, V., Emani, M., Foreman, S., Xie, Z., Lin, D., Shukla, M., Nie, W., Romero, J., Dallago, C., Vahdat, A., Xiao, C., Gibbs, T., Foster, I., Davis, J. J., Papka, M. E., Brettin, T., Stevens, R., Anandkumar, A., Vishwanath, V., & Ramanathan, A.
Affiliations
  • Argonne National Laboratory (US)
  • Harvard University (US)
  • Cerebras Systems (United States) (US)
  • Northern Illinois University (US)
  • Nvidia (United States) (US)
  • University of Illinois Urbana-Champaign (US)
Corresponding author
  • Vishwanath, Venkatram
  • Ramanathan, Arvind
Venue
The International Journal of High Performance Computing Applications
Publisher
openRxiv
Published
2022-10-11
Volume
37
Issue
6
Pages
683-705
ISSN
0890-2720 (ISSN-L)
OpenAlex
W4387994442
Cited by
120
References
57
Citations by year
2026: 30 · 2025: 40 · 2024: 35 · 2023: 14 · 2022: 1
FWCI
24.63
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • The International Journal of High Performance Computing Applications (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • Espace ÉTS (ETS) (submittedVersion)
Funders
  • National Science Foundation (DEAC02-05CH11231, DE-AC02–06CH11357, 2117896, 17-SC-20-SC)
  • U.S. Department of Energy (DE‐ AC02‐06CH11357, COVID-19, 17‐SC‐20‐SC, -AC02-05CH11231, AC02‐06CH11357, 05CH11231, DE AC02 05CH11231, 06CH11357, DE-AC02-)
  • National Energy Research Scientific Computing Center (DE–AC02–05CH11231, 05CH11231, DE-AC02-06CH11357, AC02-05CH11231)
  • National Institutes of Health (DE-AC02–05CH11231, COVID-19, DE-AC02–06CH11357, 17-SC-20-SC)
  • Office of Science (AC02‐06CH11357, 17-SC20-SC, DEAC02–05CH11231, AC02-05CH11231, DE-AC02, 05CH11231, 06CH11357, DE-AC02-06CH11357)
  • National Nuclear Security Administration (AC02-05CH11231, 17- SC-20-SC, DE-AC02-05CH11231, DE-AC02-06CH11357)
  • National Institute of Allergy and Infectious Diseases (COVID-19, P01AI165077, DE-AC02-06CH11357)
  • Argonne National Laboratory (DE- AC02-06-CH11357, DE-AC02, AC02–06CH11357, DE-AC02-05CH11231, 06CH11357)
  • Lawrence Berkeley National Laboratory (AC02‐05CH11231, DE-AC02-05CH11231, DE-AC02-06CH11357, 05CH11231)
Topics
  • Health Sciences › Medicine › Infectious Diseases › SARS-CoV-2 and COVID-19 Research
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
SARS-CoV-2, COVID-19 variants of concern, variant identification, viral evolution, foundation models, mixed-precision training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review