Light attention predicts protein location from the language of life
Open accessPublished version of a preprint
Abstract
Summary: Although knowing where a protein functions in a cell is important to characterize biological processes, this information remains unavailable for most known proteins. Machine learning narrows the gap through predictions from expert-designed input features leveraging information from multiple sequence alignments (MSAs) that is resource expensive to generate. Here, we showcased using embeddings from protein language models for competitive localization prediction without MSAs. Our lightweight deep neural network architecture used a softmax weighted aggregation mechanism with linear complexity in sequence length referred to as light attention. The method significantly outperformed the state-of-the-art (SOTA) for 10 localization classes by about 8 percentage points (Q10). So far, this might be the highest improvement of just embeddings over MSAs. Our new test set highlighted the limits of standard static datasets: while inviting new models, they might not suffice to claim improvements over the SOTA. Availability and implementation: The novel models are available as a web-service at http://embed.protein.properties. Code needed to reproduce results is provided at https://github.com/HannesStark/protein-localization. Predictions for the human proteome are available at https://zenodo.org/record/5047020. Supplementary information: Supplementary data are available at Bioinformatics Advances online.
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Integrity signals
Registry
Cites
Statistics
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Run the check — free, no account needed. It reads the open-access text and checks the sources it cites.
Claims other papers make about this one
Each is checked against this paper’s own text: does it say what the citing paper says it does?
Downstream model engineering can lead to performance improvements.
“confirming that downstream model engineering can lead to performance improvements47.”
From Nucleotide Transformer: building and evaluating robust foundation models for human genomics (2025) · cites this paper as [47]
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Cites
Hochreiter, S., & Schmidhuber, J. (1997) · Neural Computation · doi:10.1162/neco.1997.9.8.1735 · Cited by 102,318 · see citing works
Adam: A Method for Stochastic Optimization
Kingma, D. P., & Ba, J. (2014) · arXiv · doi:10.48550/arxiv.1412.6980 · Cited by 98,946 · see citing works
Gapped BLAST and PSI-BLAST: a new generation of protein database search programs
Altschul, S. (1997) · Nucleic Acids Research · doi:10.1093/nar/25.17.3389 · Cited by 73,302 · see citing works
Cortes, C., & Vapnik, V. (1995) · Machine Learning · doi:10.1007/bf00994018 · Cited by 59,421 · see citing works
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 48,293 · see citing works
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764 · see citing works
Berman, H. M. (2000) · Nucleic Acids Research · doi:10.1093/nar/28.1.235 · Cited by 40,196 · see citing works
Neural Machine Translation by Jointly Learning to Align and Translate
Bahdanau, D., Cho, K., & Bengio, Y. (2014) · arXiv · doi:10.48550/arxiv.1409.0473 · Cited by 16,213 · see citing works
Deep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983 · see citing works
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929 · see citing works
UMAP: Uniform Manifold Approximation and Projection
McInnes, L., Healy, J., Saul, N., & Großberger, L. (2018) · Journal of Open Source Software · doi:10.21105/joss.00861 · Cited by 9,826 · see citing works
SIFT: predicting amino acid changes that affect protein function
Ng, P. C. (2003) · Nucleic Acids Research · doi:10.1093/nar/gkg509 · Cited by 6,601 · see citing works
UniProt: the universal protein knowledgebase in 2021
The UniProt Consortium, Bateman, A., Martin, M.-J., Orchard, S., Magrane, M., Agivetova, R., Ahmad, S., Alpi, E., Bowler-Barnett, E. H., Britto, R., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkaa1100 · Cited by 6,372 · see citing works
Amino acid substitution matrices from protein blocks.
Henikoff, S., & Henikoff, J. G. (1992) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.89.22.10915 · Cited by 6,218 · see citing works
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · see citing works
WoLF PSORT: protein localization predictor
Horton, P., Park, K., Obayashi, T., Fujita, N., Harada, H., Adams-Collier, C. J., & Nakai, K. (2007) · Nucleic Acids Research · doi:10.1093/nar/gkm259 · Cited by 4,154 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286 · see citing works
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · see citing works
Prediction of Protein Secondary Structure at Better than 70% Accuracy
Rost, B., & Sander, C. (1993) · Journal of Molecular Biology · doi:10.1006/jmbi.1993.1413 · Cited by 2,927 · see citing works
Self-citationProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
Opportunities and obstacles for deep learning in biology and medicine
Ching, T., Himmelstein, D. S., Beaulieu-Jones, B. K., Kalinin, A. A., Do, B. T., Way, G. P., Ferrero, E., Agapow, P.-M., Zietz, M., Hoffman, M. M., et al. (2018) · Journal of The Royal Society Interface · doi:10.1098/rsif.2017.0387 · Cited by 2,310 · see citing works
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928 · see citing works
Twilight zone of protein sequence alignments
Rost, B. (1999) · Protein Engineering, Design and Selection · doi:10.1093/protein/12.2.85 · Cited by 1,833 · see citing works
Self-citationPrediction of protein subcellular localization
Yu, C., Chen, Y., Lu, C., & Hwang, J. (2006) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.21018 · Cited by 1,802 · see citing works
Database of homology‐derived protein structures and the structural meaning of sequence alignment
Sander, C., & Schneider, R. (1991) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.340090107 · Cited by 1,638 · see citing works
DeepLoc: prediction of protein subcellular localization using deep learning
Armenteros, J. J. A., Sønderby, C. K., Sønderby, S. K., Nielsen, H., & Winther, O. (2017) · Bioinformatics · doi:10.1093/bioinformatics/btx431 · Cited by 1,144 · see citing works
Unified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · see citing works
Clustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · see citing works
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · see citing works
Modeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · see citing works
Self-citationModeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606 · see citing works
Self-citationComparing two K-category assignments by a K-category correlation coefficient
Gorodkin, J. (2004) · Computational Biology and Chemistry · doi:10.1016/j.compbiolchem.2004.09.006 · Cited by 545 · see citing works
BUSCA: an integrative web server to predict subcellular localization of proteins
Savojardo, C., Martelli, P. L., Fariselli, P., et al. (2018) · Nucleic Acids Research · doi:10.1093/nar/gky320 · Cited by 452 · see citing works
Enzyme Function Less Conserved than Anticipated
Rost, B. (2002) · Journal of Molecular Biology · doi:10.1016/s0022-2836(02)00016-5 · Cited by 383 · see citing works
Self-citationBaCelLo: a balanced subcellular localization predictor
Pierleoni, A., Martelli, P. L., Fariselli, P., & Casadio, R. (2006) · Bioinformatics · doi:10.1093/bioinformatics/btl222 · Cited by 341 · see citing works
Transformer protein language models are unsupervised structure learners
Rao, R., Meier, J., Sercu, T., Ovchinnikov, S., & Rives, A. (2020) · doi:10.1101/2020.12.15.422761 · Cited by 341 · see citing works
LocTree3 prediction of localization
Goldberg, T., Hecht, M., Hamp, T., Karl, T., Yachdav, G., Ahmed, N., Altermann, U., Angerer, P., Ansorge, S., Balasz, K., et al. (2014) · Nucleic Acids Research · doi:10.1093/nar/gku396 · Cited by 320 · see citing works
Self-citationChou, K., Wu, Z., & Xiao, X. (2011) · PLoS ONE · doi:10.1371/journal.pone.0018258 · Cited by 319 · see citing works
YLoc—an interpretable web server for predicting subcellular localization
Briesemeister, S., hrer, J. ⁄. R. R. ⁄., & Kohlbacher, O. (2010) · Nucleic Acids Research · doi:10.1093/nar/gkq477 · Cited by 315 · see citing works
Blum, T., Briesemeister, S., & Kohlbacher, O. (2009) · BMC Bioinformatics · doi:10.1186/1471-2105-10-274 · Cited by 305 · see citing works
Mimicking Cellular Sorting Improves Prediction of Subcellular Localization
Nair, R. P., & Rost, B. (2005) · Journal of Molecular Biology · doi:10.1016/j.jmb.2005.02.025 · Cited by 290 · see citing works
Self-citationAutomatic prediction of protein function
Wrzeszczyński, K. O., Ofran, Y., Rost, B., et al. (2003) · Cellular and Molecular Life Sciences · doi:10.1007/s00018-003-3114-8 · Cited by 271 · see citing works
Self-citationSNAP predicts effect of mutations on protein function
Bromberg, Y., Yachdav, G., & Rost, B. (2008) · Bioinformatics · doi:10.1093/bioinformatics/btn435 · Cited by 262 · see citing works
Self-citationPredictProtein - Predicting Protein Structure and Function for 29 Years
Bernhofer, M., Dallago, C., Karl, T., Satagopam, V., Heinzinger, M., Littmann, M., Olenyi, T., Qiu, J., Schütze, K., Yachdav, G., et al. (2021) · Nucleic Acids Research · doi:10.1093/nar/gkab354 · Cited by 239 · see citing works
Self-citationLearning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · see citing works
Sequence conserved for subcellular localization
Nair, R. P., & Rost, B. (2002) · Protein Science · doi:10.1110/ps.0207402 · Cited by 175 · see citing works
Self-citationEmbeddings from deep learning transfer GO annotations beyond homology
Littmann, M., Heinzinger, M., Dallago, C., Olenyi, T., & Rost, B. (2021) · Scientific Reports · doi:10.1038/s41598-020-80786-0 · Cited by 169 · see citing works
Self-citationProtein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
Self-citationSherLoc2: A High-Accuracy Hybrid Method for Predicting Subcellular Localization of Proteins
Briesemeister, S., Blum, T., Brady, S., Lam, Y. P., Kohlbacher, O., & Shatkay, H. (2009) · Journal of Proteome Research · doi:10.1021/pr900665y · Cited by 138 · see citing works
LocTree2 predicts localization for all domains of life
Goldberg, T., Hamp, T., & Rost, B. (2012) · Bioinformatics · doi:10.1093/bioinformatics/bts390 · Cited by 114 · see citing works
Self-citationLearned Embeddings from Deep Learning to Visualize and Predict Protein Sets
Dallago, C., Schütze, K., Heinzinger, M., Olenyi, T., Littmann, M., Lu, A. X., Yang, K., Min, S., Yoon, S., Morton, J. T., et al. (2021) · Current Protocols · doi:10.1002/cpz1.113 · Cited by 101 · see citing works
Self-citationHFSP: high speed homology-driven function annotation of proteins
Mahlich, Y., Steinegger, M., Rost, B., & Bromberg, Y. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty262 · Cited by 42 · see citing works
Self-citationSingle Layers of Attention Suffice to Predict Protein ContactsPreprint
Bhattacharya, N., Thomas, N., Rao, R., Dauparas, J., Koo, P. K., Baker, D. A., Song, Y. S., & Ovchinnikov, S. (2020) · bioRxiv (Cold Spring Harbor Laboratory) · doi:10.1101/2020.12.21.423882 · Cited by 31 · see citing works
SChloro: directing Viridiplantae proteins to six chloroplastic sub-compartments
Savojardo, C., Martelli, P. L., Fariselli, P., et al. (2016) · Bioinformatics · doi:10.1093/bioinformatics/btw656 · Cited by 22 · see citing works
Protein profiles: Biases and protocols
Urban, G., Torrisi, M., Magnan, C. N., et al. (2020) · Computational and Structural Biotechnology Journal · doi:10.1016/j.csbj.2020.08.015 · Cited by 12 · see citing works
Spectrum of Protein Location in Proteomes Captures Evolutionary Relationship Between Species
Marot-Lassauzaie, V., Goldberg, T., Armenteros, J. J. A., et al. (2021) · Journal of Molecular Evolution · doi:10.1007/s00239-021-10022-4 · Cited by 2 · see citing works
Flagged first, then by citations
Cited by
Sorted by citations
ProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · see citing works
Nucleotide Transformer: building and evaluating robust foundation models for human genomics
Dalla-Torre, H., Gonzalez, L., Mendoza-Revilla, J., Lopez Carranza, N., Grzywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., de Almeida, B. P., Sirelkhatim, H., et al. (2025) · Nature Methods · doi:10.1038/s41592-024-02523-z · Cited by 656 · see citing works
“In comparison to the extensive probing exercise, this technique yielded superior results while using fewer compute resources, confirming that downstream model engineering can lead to performance improvements 47 .”
Self-citationTransfer learning to leverage larger datasets for improved prediction of protein stability changes
Dieckhaus, H., Brocidiacono, M., Randolph, N. Z., & Kuhlman, B. (2024) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2314853121 · Cited by 205 · see citing works
Protein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139 · see citing works
Self-citationNovel machine learning approaches revolutionize protein knowledge
Bordin, N., Dallago, C., Heinzinger, M., Kim, S., Littmann, M., Rauer, C., Steinegger, M., Rost, B., & Orengo, C. (2023) · Trends in Biochemical Sciences · doi:10.1016/j.tibs.2022.11.001 · Cited by 72 · see citing works
Self-citationFeature Reuse and Scaling: Understanding Transfer Learning with Protein Language Models
Li, F.-Z., Amini, A. P., Yue, Y., Yang, K. K., & Lu, A. X. (2024) · doi:10.1101/2024.02.05.578959 · Cited by 51 · see citing works
- Authors
- Stärk, H., Dallago, C., Heinzinger, M., & Rost, B.
- Affiliations
- Technical University of Munich (DE)
- Corresponding author
- Dallago, Christian
- Rost, Burkhard
- Venue
- Bioinformatics Advances
- Publisher
- Oxford University Press (OUP)
- Published
- 2021-01-01
- Volume
- 1
- Issue
- 1
- Pages
- vbab035-vbab035
- ISSN
- 2635-0041 (ISSN-L)
- PMC
- PMC9710637
- OpenAlex
- W4242765109
- Cited by
- 130
- References
- 57
- Citations by year
- 2026: 16 · 2025: 32 · 2024: 33 · 2023: 23 · 2022: 20 · 2021: 4 · 2020: 2
- FWCI
- 6.28
- Type
- journal article
- Language
- en
- Open access
- gold
- doi.org
- Locations
- Bioinformatics Advances (publishedVersion, cc-by)
- bioRxiv (Cold Spring Harbor Laboratory) (acceptedVersion, cc-by-nd)
- PubMed Central (publishedVersion, cc-by)
- PubMed (publishedVersion)
- mediaTUM – the media and publications repository of the Technical University Munich (Technical University Munich) (submittedVersion, other-oa)
- Funders
- Deutsche Forschungsgemeinschaft (RO1320/4-1)
- Bundesministerium für Bildung und Forschung (031L0168)
- BMBF through the program ‘Software Campus 2.0 (TU München)’ (01IS17049)
- Topics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Machine Learning in Bioinformatics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Protein Structure and Dynamics
- Life Sciences › Biochemistry, Genetics and Molecular Biology › Molecular Biology › Genomics and Phylogenetic Studies
- Keywords
- protein localization prediction, protein language models, protein embeddings, human proteome, benchmark test set
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
