Convolutions are competitive with transformers for protein sequence pretraining
Open accessPublished version of a preprint
Abstract
Pretrained protein sequence language models have been shown to improve the performance of many prediction tasks and are now routinely integrated into bioinformatics tools. However, these models largely rely on the transformer architecture, which scales quadratically with sequence length in both run-time and memory. Therefore, state-of-the-art models have limitations on sequence length. To address this limitation, we investigated whether convolutional neural network (CNN) architectures, which scale linearly with sequence length, could be as effective as transformers in protein language models. With masked language model pretraining, CNNs are competitive with, and occasionally superior to, transformers across downstream applications while maintaining strong performance on sequences longer than those allowed in the current state-of-the-art transformer models. Our work suggests that computational efficiency can be improved without sacrificing performance, simply by using a CNN architecture instead of a transformer, and emphasizes the importance of disentangling pretraining task and model architecture. A record of this paper's transparent peer review process is included in the supplemental information.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Evo reaches performance competitive with leading protein-specific language models on bacterial proteins.
“Evo also reaches competitive performance with leading protein-specific language models (41, 46–48) (Fig. 2B).”
From Sequence modeling and design from molecular to genome scale with Evo (2024) · cites this paper as [46]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Advances in neural information processing systems 7
(1996) · Computers & Mathematics with Applications · doi:10.1016/s0898-1221(96)90463-0 · Cited by 14,759 · see citing works
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Longformer: The Long-Document Transformer
Beltagy, I., Peters, M. E., & Cohan, A. (2020) · arXiv · doi:10.48550/arxiv.2004.05150 · Cited by 2,281 · see citing works
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928 · see citing works
Reformer: The Efficient Transformer
Kitaev, N., Kaiser, Ł., & Levskaya, A. (2020) · arXiv · doi:10.48550/arxiv.2001.04451 · Cited by 1,435 · see citing works
Improved protein structure prediction using predicted interresidue orientations
Yang, J., Anishchenko, I., Park, H., Peng, Z., Ovchinnikov, S., & Baker, D. (2020) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.1914677117 · Cited by 1,421 · see citing works
Generating Long Sequences with Sparse Transformers
Child, R., Gray, S., Radford, A., & Sutskever, I. (2019) · arXiv · doi:10.48550/arxiv.1904.10509 · Cited by 1,063 · see citing works
ProteinBERT: a universal deep-learning model of protein sequence and function
Brandes, N., Ofer, D., Peleg, Y., Rappoport, N., & Linial, M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac020 · Cited by 1,009 · see citing works
Linformer: Self-Attention with Linear Complexity
Wang, S., Li, B. Z., Khabsa, M., Fang, H., & Ma, H. (2020) · arXiv · doi:10.48550/arxiv.2006.04768 · Cited by 867 · see citing works
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
Katharopoulos, A., Vyas, A., Pappas, N., & Fleuret, F. (2020) · arXiv · doi:10.48550/arxiv.2006.16236 · Cited by 827 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Rethinking Attention with Performers
Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. (2020) · arXiv · doi:10.48550/arxiv.2009.14794 · Cited by 695 · see citing works
Pay Less Attention with Lightweight and Dynamic ConvolutionsPreprint
Wu, F., Fan, A., Baevski, A., Dauphin, Y., & Auli, M. (2019) · International Conference on Learning Representations · doi:10.48550/arxiv.1901.10430 · Cited by 669 · see citing works
Deep generative models of genetic variation capture the effects of mutations
Riesselman, A. J., Ingraham, J. B., & Marks, D. S. (2018) · Nature Methods · doi:10.1038/s41592-018-0138-4 · Cited by 601 · see citing works
Local fitness landscape of the green fluorescent protein
Sarkisyan, K. S., Bolotin, D. A., Meer, M. V., Usmanova, D. R., Mishin, A. S., Sharonov, G. V., Ivankov, D. N., Bozhanova, N. G., Baranov, M. S., Soylemez, O., et al. (2016) · Nature · doi:10.1038/nature17995 · Cited by 591 · see citing works
Global analysis of protein folding using massively parallel design, synthesis, and testing
Rocklin, G. J., Chidyausiku, T. M., Goreshnik, I., Ford, A., Houliston, S., Lemak, A., Carter, L., Ravichandran, R., Mulligan, V. K., Chevalier, A., et al. (2017) · Science · doi:10.1126/science.aan0693 · Cited by 501 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338 · see citing works
Using deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332 · see citing works
Protein design and variant prediction using autoregressive generative models
Shin, J.-E., Riesselman, A. J., Kollasch, A. W., McMahon, C., Simon, E., Sander, C., Manglik, A., Kruse, A. C., & Marks, D. S. (2021) · Nature Communications · doi:10.1038/s41467-021-22732-w · Cited by 320 · see citing works
Deep diversification of an AAV capsid protein by machine learning
Bryant, D., Bashir, A., Sinai, S., Jain, N. K., Ogden, P. J., Riley, P. F., Church, G. M., Colwell, L. J., & Kelsic, E. D. (2021) · Nature Biotechnology · doi:10.1038/s41587-020-00793-4 · Cited by 302 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.03.07.982272 · Cited by 282 · see citing works
Adaptation in protein fitness landscapes is facilitated by indirect paths
Wu, N. C., Dai, L., Olson, C. A., Lloyd‐Smith, J. O., & Sun, R. X. (2016) · eLife · doi:10.7554/elife.16965 · Cited by 281 · see citing works
Meltome atlas—thermal proteome stability across the tree of life
Jarząb, A., Kurzawa, N., Hopf, T. A., Moerch, M., Zecha, J., Leijten, N. M., Bian, Y., Musiol, E., Maschberger, M., Stoehr, G., et al. (2020) · Nature Methods · doi:10.1038/s41592-020-0801-4 · Cited by 275 · see citing works
Long Range Arena: A Benchmark for Efficient Transformers
Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., & Metzler, D. (2020) · arXiv · doi:10.48550/arxiv.2011.04006 · Cited by 241 · see citing works
Ankh ☥: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., Moustafa, W., Elkerdawy, M., Rochereau, C., & Rost, B. (2023) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2023.01.16.524265 · Cited by 201 · see citing works
Haas, J., Barbato, A., Behringer, D., Studer, G., Roth, S., Bertoni, M., Mostaguir, K., Gumienny, R., & Schwede, T. (2018) · Proteins: Structure, Function, and Bioinformatics · doi:10.1002/prot.25431 · Cited by 168 · see citing works
Tailoring the AAV vector capsid for gene therapy
Vandenberghe, L. H., Wilson, J. M., & Gao, G. (2009) · Gene Therapy · doi:10.1038/gt.2008.170 · Cited by 155 · see citing works
Hie, B., Yang, K., & Kim, P. S. (2022) · Cell Systems · doi:10.1016/j.cels.2022.01.003 · Cited by 147 · see citing works
Self-citationEngineering the AAV capsid to optimize vector–host-interactions
Büning, H., Huber, A., Zhang, L. N., Meumann, N., & Hacker, U. T. (2015) · Current Opinion in Pharmacology · doi:10.1016/j.coph.2015.08.002 · Cited by 113 · see citing works
Zarin, T., Strome, B., Peng, G., Pritišanac, I., Forman-Kay, J. D., & Moses, A. M. (2021) · eLife · doi:10.7554/elife.60220 · Cited by 113 · see citing works
Engineering the AAV capsid to evade immune responses
Barnes, C., Scheideler, O. J., & Schaffer, D. V. (2019) · Current Opinion in Biotechnology · doi:10.1016/j.copbio.2019.01.002 · Cited by 105 · see citing works
Are Pretrained Convolutions Better than Pretrained Transformers?
Tay, Y., Dehghani, M., Gupta, J. P., Aribandi, V., Bahri, D., Qin, Z., & Metzler, D. (2021) · Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2021.acl-long.335 · Cited by 87 · see citing works
Wang, Z., Combs, S. A., Brand, R. M., Calvo, M. R., Xu, P., Price, G., Golovach, N., Salawu, E. O., Wise, C., Ponnapalli, S. P., et al. (2022) · Scientific Reports · doi:10.1038/s41598-022-10775-y · Cited by 84 · see citing works
Masked Language Modeling for Proteins via Linearly Scalable Long-Context TransformersPreprint
Choromański, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlós, T., Hawkins, P., Davis, J. Q., Belanger, D., Colwell, L. J., et al. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2006.03555 · Cited by 83 · see citing works
Self-Supervised Contrastive Learning of Protein Representations By Mutual Information MaximizationPreprint
Lu, A. X., Zhang, H., Ghassemi, M., & Moses, A. M. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.09.04.283929 · Cited by 70 · see citing works
Self-citationAssessing the accuracy of contact predictions in CASP13
Shrestha, R., Fajardo, E. J., Gil, N., Fidelis, K. A., Kryshtafovych, A., Monastyrskyy, B., & Fiser, A. (2019) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25819 · Cited by 62 · see citing works
Deep mutational scanning of S. pyogenes Cas9 reveals important functional domains
Spencer, J. M., & Zhang, X. (2017) · Scientific Reports · doi:10.1038/s41598-017-17081-y · Cited by 61 · see citing works
Structure-aware protein self-supervised learning
Chen, C., Zhou, J., Wang, F., Liu, X., & Dou, D. (2023) · Bioinformatics · doi:10.1093/bioinformatics/btad189 · Cited by 54 · see citing works
Is Transfer Learning Necessary for Protein Landscape Prediction?Preprint
Shanehsazzadeh, A., Belanger, D., & Dohan, D. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2011.03443 · Cited by 43 · see citing works
ProGen: Language Modeling for Protein GenerationPreprint
Madani, A., McCann, B., Naik, N., Keskar, N. S., Anand, N., Eguchi, R. R., Huang, P., & Socher, R. (2020) · arXiv (Cornell University) · doi:10.48550/arxiv.2004.03497 · Cited by 42 · see citing works
Are Pre-trained Convolutions Better than Pre-trained Transformers?Preprint
Tay, Y., Dehghani, M., Gupta, J. P., Bahri, D., Aribandi, V., Qin, Z., & Metzler, D. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2105.03322 · Cited by 33 · see citing works
Ankh: Optimized Protein Language Model Unlocks General-Purpose ModellingPreprint
Elnaggar, A., Essam, H., Salah-Eldin, W., et al. (2023) · arXiv (Cornell University) · doi:10.48550/arxiv.2301.06568 · Cited by 31 · see citing works
Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language ModelPreprint
He, L., Zhang, S., Wu, L., Xia, H., Ju, F., Zhang, H., Liu, S., Xia, Y., Zhu, J., Deng, P., et al. (2021) · arXiv (Cornell University) · doi:10.48550/arxiv.2110.15527 · Cited by 20 · see citing works
A Deep SE(3)-Equivariant Model for Learning Inverse Protein FoldingPreprint
McPartlon, M., Lai, B., & Xu, J. (2022) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2022.04.15.488492 · Cited by 18 · see citing works
Mansoor, S., Baek, M., Madan, U., & Horvitz, E. (2021) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2021.09.01.458592 · Cited by 12 · see citing works
Flagged first, then by citations
Research works printed but not linked: 6
- Rao, R.M., Liu, J., Verkuil, R., Meier, J., Canny, J., Abbeel, P., Sercu, T., and Rives, A. (2021). MSA transformer. In Proceedings of the 38th International Conference on Machine Learning PMLR, 139, M. Meila and T. Zhang, eds., pp. 8844-8856. https://proceedings.mlr.press/v139/ rao21a.html.
- Dallago, C., Mou, J., Johnston, KadinaE., Wittmann, B., Bhattacharya, N., Goldman, S., Madani, A., and Yang, K.K. (2021). FLIP: benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
- Tay, Y., Bahri, D., Yang, L., Metzler, D., and Juan, D.-C. (2020a). Sparse sinkhorn attention. In International Conference on Machine Learning PMLR, pp. 9438-9447.
- Zaheer, M., Guruganesh, G., Dubey, K.A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. (2020). Big bird: trans- formers for longer sequences. Adv. Neural Inf. Process. Syst. 33, 17283-17297.
- Zhang, Zuobai, Xu, M., Jamasb, A.R., Chenthamarakshan, V., Lozano, A., Das, P., and Tang, J. (2023). Protein representation learning by geometric structure pretraining. In The Eleventh International Conference on Learning Representations https://openreview.net/forum?id=to3qCB3tOh9.
- Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. (2022). Learning inverse folding from millions of predicted structures. In International Conference on Machine Learning PMLR, pp. 8946-8970.
Cited by
Sorted by citations
Evolutionary-scale prediction of atomic level protein structure with a language model
Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. (2022) · doi:10.1101/2022.07.20.500902 · Cited by 5,523 · see citing works
ProtGPT2 is a deep unsupervised language model for protein design
Ferruz, N., Schmidt, S., & Höcker, B. (2022) · Nature Communications · doi:10.1038/s41467-022-32007-7 · Cited by 795 · see citing works
Sequence modeling and design from molecular to genome scale with Evo
Nguyen, E., Poli, M., Durrant, M. G., Kang, B., Katrekar, D., Li, D. B., Bartie, L. J., Thomas, A. W., King, S. H., Brixi, G., et al. (2024) · Science · doi:10.1126/science.ado9336 · Cited by 580 · see citing works
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480 · see citing works
DNA language models are powerful predictors of genome-wide variant effects
Benegas, G., Batra, S. S., & Song, Y. S. (2023) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2311219120 · Cited by 157 · see citing works
Feature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., & Lu, A. X. (2024) · doi:10.1101/2024.02.05.578959 · Cited by 51 · see citing works
Self-citation
- Authors
- Yang, K. K., Fusi, N., & Lu, A. X.
- Affiliations
- Microsoft Research New England (United States) (US)
- Corresponding author
- Yang, Kevin K.
- Venue
- Cell Systems
- Publisher
- Elsevier BV
- Published
- 2024-03
- Volume
- 15
- Issue
- 3
- Pages
- 286-294.e2
- ISSN
- 2405-4712 (ISSN-L)
- OpenAlex
- W4392302569
- Cited by
- 137
- References
- 48
- Citations by year
- 2026: 28 · 2025: 46 · 2024: 36 · 2023: 22 · 2022: 5
- FWCI
- 15.00
- Type
- journal article
- Language
- en
- Open access
- bronze
- doi.org
- Licence
- www.elsevier.com/tdm/userlicense/1.0/ (tdm)
- www.elsevier.com/legal/tdmrep-license (tdm)
- www.elsevier.com/open-access/userlicense/1.0/ (vor, 384 d)
- doi.org/10.15223/policy-017 (stm-asf)
- doi.org/10.15223/policy-037 (stm-asf)
- doi.org/10.15223/policy-012 (stm-asf)
- doi.org/10.15223/policy-029 (stm-asf)
- doi.org/10.15223/policy-004 (stm-asf)
- Locations
- Cell Systems (publishedVersion)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, public-domain)
- PubMed (publishedVersion)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- convolutional neural networks, transformer models, computational efficiency, protein sequence prediction
- MeSH terms
- Amino Acid Sequence, Peer Review, Neural Networks, Computer, Computational Biology
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
