CoChat
Sign Up
PreprintCited by 123SARS-CoV-2 and COVID-19 ResearchGenomics and Phylogenetic StudiesMachine Learning in Bioinformatics

The DOI matches its Crossref record; 57 of the 57 references it cites were found in a registry or an index.

GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics

Open access
Zvyagin, Maxim, Brace, Alexander, Hippe, Kyle, Deng, Yuntian, Zhang, Bin, Bohorquez, Cindy Orozco, Clyde, Austin, Kale, Bharat, Perez-Rivera, Danilo, Ma, Heng, Mann, Carla M., Irvin, Michael, Gregory Pauloski, J., Ward, Logan, Hayot-Sasson, Valerie, Emani, Murali, Foreman, Sam, Xie, Zhen, Lin, Diangen, Shukla, Maulik, Nie, Weili, Romero, Josh, Dallago, Christian, Vahdat, Arash, Xiao, Chaowei, Gibbs, Thomas, Foster, Ian, Davis, James J., Papka, Michael E., Brettin, Thomas, Stevens, Rick, Anandkumar, Anima, Vishwanath, Venkatram, Ramanathan, Arvind·2022·The International Journal of High Performance Computing Applications
DOI 10.1101/2022.10.10.511571PMC PMC9709791

Preprint — published as 10.1177/10943420231201154 · Paper page

Open this paper in CoChat — free accountOpen PDF

Abstract

We seek to transform how new and emergent variants of pandemiccausing viruses, specifically SARS-CoV-2, are identified and classified. By adapting large language models (LLMs) for genomic data, we build genome-scale language models (GenSLMs) which can learn the evolutionary landscape of SARS-CoV-2 genomes. By pretraining on over 110 million prokaryotic gene sequences and finetuning a SARS-CoV-2-specific model on 1.5 million genomes, we show that GenSLMs can accurately and rapidly identify variants of concern. Thus, to our knowledge, GenSLMs represents one of the first whole genome scale foundation models which can generalize to other prediction tasks. We demonstrate scaling of GenSLMs on GPU-based supercomputers and AI-hardware accelerators utilizing 1.63 Zettaflops in training runs with a sustained performance of 121 PFLOPS in mixed precision and peak of 850 PFLOPS. We present initial scientific insights from examining GenSLMs in tracking evolutionary dynamics of SARS-CoV-2, paving the path to realizing this on large biological data.

Overview

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Integrity signals

Registry

Crossref · 10.1101/2022.10.10.511571

Cites

57 of 57 references verified0 citation errors · 0 not checkable

Statistics

Figures

Figure 1: Overview of GenSLM models for predictive modeling of SARS-CoV-2 evolution. The inputs to GenSLM are nucleotide sequences, encoded at the codon level (every three nucleotide represents a codon; hence the 20 natural amino acid language is described by 64 codons). These inputs are successively fed into transformer blocks (referred to as layers (Li)), which ultimately results in learning a semantic embedding z→ space from which one may obtain the probability of any given sequence token p(X→i∣X→[N]\{i}), where N represents the sequence length and 𝑖 represents a particular position in the entire genome.
Figure 1: Overview of GenSLM models for predictive modeling of SARS-CoV-2 evolution. The inputs to GenSLM are nucleotide sequences, encoded at the codon level (every three nucleotide represents a codon; hence the 20 natural amino acid language is described by 64 codons). These inputs are successively fed into transformer blocks (referred to as layers (Li)), which ultimately results in learning a semantic embedding z→ space from which one may obtain the probability of any given sequence token p(X→i∣X→[N]\{i}), where N represents the sequence length and 𝑖 represents a particular position in the entire genome.CC BY-NC-ND · PubMed Central
Figure 2: GenSLMs learned latent space describes biologically meaningful properties for SARS-CoV-2 genomes. (A) The embeddings from GenSLMs are visualized with t-distributed stochastic neighbor embedding (t-SNE) and each gene sequence is represented as a dot in the 2D plot. We paint each sequence by its variant ID – although we have more than 515 PANGO (Rambaut et al., 2020) lineages represented in the data, we only show those with WHO designated labels. (B) The latent space can also be painted with the MAFFT-determined alignment score (Yamada et al., 2016) with respect to an Omicron genome; clustering in the distance measures is clearly visible. Visualizing the sequence log-likelihood (blue bar) and the cross-protein attention (orange lines) from (C) Delta and (D) Omicron SARS-CoV-2 strains highlights how different the co-evolutionary patterns are in these lineages. It is interesting to note that while the Spike protein from Delta strain shows coupling to nsp3, nsp5, and other proteins, these couplings are not observed in the Omicron strain.
Figure 2: GenSLMs learned latent space describes biologically meaningful properties for SARS-CoV-2 genomes. (A) The embeddings from GenSLMs are visualized with t-distributed stochastic neighbor embedding (t-SNE) and each gene sequence is represented as a dot in the 2D plot. We paint each sequence by its variant ID – although we have more than 515 PANGO (Rambaut et al., 2020) lineages represented in the data, we only show those with WHO designated labels. (B) The latent space can also be painted with the MAFFT-determined alignment score (Yamada et al., 2016) with respect to an Omicron genome; clustering in the distance measures is clearly visible. Visualizing the sequence log-likelihood (blue bar) and the cross-protein attention (orange lines) from (C) Delta and (D) Omicron SARS-CoV-2 strains highlights how different the co-evolutionary patterns are in these lineages. It is interesting to note that while the Spike protein from Delta strain shows coupling to nsp3, nsp5, and other proteins, these couplings are not observed in the Omicron strain.CC BY-NC-ND · PubMed Central
Figure 3: Illustration of diffusion-based hierarchical modeling. To predict a codon (such as TAA), we use both the previous codons within the context window (we use size 3 shown in green for illustration) and the high-level representations z.
Figure 3: Illustration of diffusion-based hierarchical modeling. To predict a codon (such as TAA), we use both the previous codons within the context window (we use size 3 shown in green for illustration) and the high-level representations z.CC BY-NC-ND · PubMed Central
Figure 4: Diffusion-based hierarchical modeling of SARS-CoV-2 genomes results in generation of sequences that captures the correct context of various open reading frames (ORFs). (A) Comparison of statistics measured on generated sequences and on real data for the ORFs. Diffusion-based hierarchical LM has a global high-level plan whereas the baseline can only take into account the previous 1023 codons. (B) Generated sequences (light blue) from the model overlaid on the phylogenetic tree demonstrate that these sequences are similar to observed strains.
Figure 4: Diffusion-based hierarchical modeling of SARS-CoV-2 genomes results in generation of sequences that captures the correct context of various open reading frames (ORFs). (A) Comparison of statistics measured on generated sequences and on real data for the ORFs. Diffusion-based hierarchical LM has a global high-level plan whereas the baseline can only take into account the previous 1023 codons. (B) Generated sequences (light blue) from the model overlaid on the phylogenetic tree demonstrate that these sequences are similar to observed strains.CC BY-NC-ND · PubMed Central
Figure 5: Conceptual overview of our workflow. A “Thinker” orchestrates data flow between two applications, namely the sequence generator and the Bayesian optimization to drive the generated sequences towards a target property using reward-guided beam search, where μ represents the mixing constant used to balance the reward function against the log likelihood of generating the next token.
Figure 5: Conceptual overview of our workflow. A “Thinker” orchestrates data flow between two applications, namely the sequence generator and the Bayesian optimization to drive the generated sequences towards a target property using reward-guided beam search, where μ represents the mixing constant used to balance the reward function against the log likelihood of generating the next token.CC BY-NC-ND · PubMed Central
Figure 6: (A) Scaling results on Polaris and Selene systems for MSL=2048; (B) Scaling behavior of DDP vs. DeepSpeed runs on Selene (C) Scaling results on Polaris and Selene systems for MSL=10240;
Figure 6: (A) Scaling results on Polaris and Selene systems for MSL=2048; (B) Scaling behavior of DDP vs. DeepSpeed runs on Selene (C) Scaling results on Polaris and Selene systems for MSL=10240;CC BY-NC-ND · PubMed Central
Figure 7: Workflow utilization measured by the number of active workers (applications actively serving requests) as a function of workflow runtime measured on 224 nodes of Polaris (896 A100 GPUs). The warm-able application design realizes 97% utilization, enabling 1.9X more sequences to be generated compared to a cold start baseline.
Figure 7: Workflow utilization measured by the number of active workers (applications actively serving requests) as a function of workflow runtime measured on 224 nodes of Polaris (896 A100 GPUs). The warm-able application design realizes 97% utilization, enabling 1.9X more sequences to be generated compared to a cold start baseline.CC BY-NC-ND · PubMed Central

Verification

Claims the paper makes

The claims and statistics of this paper haven’t been checked yet.

Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.

A few free checks a day · papers already verified open instantly · results are public

Self-check

Percentages, sums and reported differences re-computed from the paper’s own numbers.

The self-check needs the paper’s full text.

Citations

Cites

Research works printed but not linked: 4

  1. Jacob Devlin , Ming-Wei Chang , Kenton Lee , and Kristina Toutanova . 2018. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprintarXiv:1810.04805 (2018).
  2. 10.1101/2020.03.21.990770 · DOI
  3. Philipp Moritz , Robert Nishihara , Stephanie Wang , Alexey Tumanov , Richard Liaw , Eric Liang , Melih Elibol , Zongheng Yang , William Paul , Michael I. Jordan , and Ion Stoica . 2018. Ray: A Distributed Framework for Emerging AI Applications. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). USENIX Association, Carlsbad, CA, 561–577. https://www.usenix.org/conference/osdi18/presentation/moritz
  4. Rose E Wang , Esin Durmus , Noah Goodman , and Tatsunori Hashimoto . 2022. Language modeling via stochastic processes. In International Conference on Learning Representations.

Cited by

Sorted by citations

See all 123

Related

See all

Metadata

Authors
Zvyagin, M., Brace, A., Hippe, K., Deng, Y., Zhang, B., Bohorquez, C. O., Clyde, A., Kale, B., Perez-Rivera, D., Ma, H., Mann, C. M., Irvin, M., Gregory Pauloski, J., Ward, L., Hayot-Sasson, V., Emani, M., Foreman, S., Xie, Z., Lin, D., Shukla, M., Nie, W., Romero, J., Dallago, C., Vahdat, A., Xiao, C., Gibbs, T., Foster, I., Davis, J. J., Papka, M. E., Brettin, T., Stevens, R., Anandkumar, A., Vishwanath, V., & Ramanathan, A.
Affiliations
  • Argonne National Laboratory (US)
  • Harvard University (US)
  • Cerebras Systems (United States) (US)
  • Northern Illinois University (US)
  • Nvidia (United States) (US)
  • University of Illinois Urbana-Champaign (US)
Corresponding author
  • Vishwanath, Venkatram
  • Ramanathan, Arvind
ORCID
Venue
The International Journal of High Performance Computing Applications
Publisher
openRxiv
Published
2022-10-11
Volume
37
Issue
6
Pages
683-705
ISSN
0890-2720 (ISSN-L)
DOI
10.1101/2022.10.10.511571
PMC
PMC9709791
OpenAlex
W4387994442
Cited by
120
References
57
Citations by year
2026: 30 · 2025: 40 · 2024: 35 · 2023: 14 · 2022: 1
FWCI
24.63
Type
journal article
Language
en
Open access
Licence
cc-by-nc-nd
Locations
  • The International Journal of High Performance Computing Applications (publishedVersion)
  • bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nc-nd)
  • PubMed (publishedVersion)
  • OSTI OAI (U.S. Department of Energy Office of Scientific and Technical Information) (submittedVersion)
  • PubMed Central (submittedVersion, cc-by-nc-nd)
  • Espace ÉTS (ETS) (submittedVersion)
Funders
  • National Science Foundation (DEAC02-05CH11231, DE-AC02–06CH11357, 2117896, 17-SC-20-SC)
  • U.S. Department of Energy (DE‐ AC02‐06CH11357, COVID-19, 17‐SC‐20‐SC, -AC02-05CH11231, AC02‐06CH11357, 05CH11231, DE AC02 05CH11231, 06CH11357, DE-AC02-)
  • National Energy Research Scientific Computing Center (DE–AC02–05CH11231, 05CH11231, DE-AC02-06CH11357, AC02-05CH11231)
  • National Institutes of Health (DE-AC02–05CH11231, COVID-19, DE-AC02–06CH11357, 17-SC-20-SC)
  • Office of Science (AC02‐06CH11357, 17-SC20-SC, DEAC02–05CH11231, AC02-05CH11231, DE-AC02, 05CH11231, 06CH11357, DE-AC02-06CH11357)
  • National Nuclear Security Administration (AC02-05CH11231, 17- SC-20-SC, DE-AC02-05CH11231, DE-AC02-06CH11357)
  • National Institute of Allergy and Infectious Diseases (COVID-19, P01AI165077, DE-AC02-06CH11357)
  • Argonne National Laboratory (DE- AC02-06-CH11357, DE-AC02, AC02–06CH11357, DE-AC02-05CH11231, 06CH11357)
  • Lawrence Berkeley National Laboratory (AC02‐05CH11231, DE-AC02-05CH11231, DE-AC02-06CH11357, 05CH11231)
Topics
  • Health Sciences › Medicine › Infectious Diseases › SARS-CoV-2 and COVID-19 Research
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
  • Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
Keywords
SARS-CoV-2, COVID-19 variants of concern, variant identification, viral evolution, foundation models, mixed-precision training

About these checks

Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.

Statistics and arithmetic are recomputed, not judged.

Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.

What counts as a claim ›How we measure confidence ›How we check this ›

Are you an author, or do you see a problem with this page? Request a review