Nearest neighbor search on embeddings rapidly identifies distant protein relations
Open accessThe DOI matches its Crossref record; 31 of the 31 references it cites were found in a registry or an index.
Abstract
Since 1992, all state-of-the-art methods for fast and sensitive identification of evolutionary, structural, and functional relations between proteins (also referred to as “homology detection”) use sequences and sequence-profiles (PSSMs). Protein Language Models (pLMs) generalize sequences, possibly capturing the same constraints as PSSMs, e.g., through embeddings. Here, we explored how to use such embeddings for nearest neighbor searches to identify relations between protein pairs with diverged sequences (remote homology detection for levels of <20% pairwise sequence identity, PIDE). While this approach excelled for proteins with single domains, we demonstrated the current challenges applying this to multi-domain proteins and presented some ideas how to overcome existing limitations, in principle. We observed that sufficiently challenging data set separations were crucial to provide deeply relevant insights into the behavior of nearest neighbor search when applied to the protein embedding space, and made all our methods readily available for others.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
Highly accurate protein structure prediction with AlphaFold
Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03819-2 · Cited by 43,764
Self-citationDeep Contextualized Word Representations
Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., & Zettlemoyer, L. (2018) · Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) · doi:10.18653/v1/n18-1202 · Cited by 11,983
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
ColabFold - Making protein folding accessible to all
Mirdita, M., Schütze, K., Moriwaki, Y., Heo, L., Ovchinnikov, S., & Steinegger, M. (2021) · doi:10.1101/2021.08.15.456425 · Cited by 9,391
Self-citationSensitive protein alignments at tree-of-life scale using DIAMOND
Buchfink, B., Reuter, K., & Drost, H.-G. (2021) · Nature Methods · doi:10.1038/s41592-021-01101-x · Cited by 5,097
MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets
Steinegger, M., & Söding, J. (2017) · Nature Biotechnology · doi:10.1038/nbt.3988 · Cited by 4,840 · cited 7×
“This was most prominently exempli fi ed by the 3.6% of ground truth homologs which knnProtT5 found but which could not be reasonably aligned [neither by MMseqs2 (Steinegger and Söding, 2017) nor by Smith-Waterman (Smith and Waterman, 1981)].”
BackgroundMethodInfluentialSelf-citationThe Pfam protein families database in 2019
El-Gebali, S., Mistry, J., Bateman, A., Eddy, S. R., Luciani, A., Potter, S. C., Qureshi, M., Richardson, L. J., Salazar, G. A., Smart, A., et al. (2019) · Nucleic Acids Research · doi:10.1093/nar/gky995 · Cited by 4,602 · cited 1×
“The Pfam database (El-Gebali et al., 2019) collects protein families without using 3D structure information.”
MethodUniProt: the universal protein knowledgebase
The UniProt Consortium (2017) · Nucleic Acids Research · doi:10.1093/nar/gkw1099 · Cited by 4,436
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jegou, H. (2021) · IEEE Transactions on Big Data · doi:10.1109/tbdata.2019.2921572 · Cited by 4,228
Billion-scale similarity search with GPUs
Johnson, J., Douze, M., & Jégou, H. (2017) · arXiv · doi:10.48550/arxiv.1702.08734 · Cited by 4,140 · cited 2×
“The k-nn search was performed using the python interface of faiss version 1.6.3 (Johnson et al., 2019).”
MethodInfluentialRives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 12×
“For instance, only the more recent ESM (Rives et al., 2021) and ProtT5 (Elnaggar et al., 2021) outperformed MMseqs2 for CATH20, while slightly earlier models such as SeqVec, or ProtBERT failed to do so.”
BackgroundMethodInfluentialHighly accurate protein structure prediction for the human proteome
Tunyasuvunakool, K., Adler, J., Wu, Z., Green, T., Zielinski, M., Žídek, A., Bridgland, A., Cowie, A., Meyer, C., Laydon, A., et al. (2021) · Nature · doi:10.1038/s41586-021-03828-1 · Cited by 3,025
Product Quantization for Nearest Neighbor Search
Jégou, H., Douze, M., & Schmid, C. (2011) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2010.57 · Cited by 2,956 · cited 2×
“Modern indexing techniques such as Hierarchical Navigable Small World Graphs (Malkov and Yashunin, 2018) or Product Quantization (Jegou et al., 2010), as well as, approaches building upon those two (Babenko and Lempitsky, 2014; Baranchuk et al., 2018; Matsui et al., 2018) handle billion-scale datasets, suggesting the…”
MethodProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329
HHblits: lightning-fast iterative protein sequence searching by HMM-HMM alignment
Remmert, M., Biegert, A., Hauser, A., & Söding, J. (2012) · Nature Methods · doi:10.1038/nmeth.1818 · Cited by 2,288
UniRef clusters: a comprehensive and scalable alternative for improving sequence similarity searches
Suzek, B. E., Wang, Y., Huang, H., McGarvey, P. B., Wu, C. H., & the UniProt Consortium (2015) · Bioinformatics · doi:10.1093/bioinformatics/btu739 · Cited by 1,928
Malkov, Y. A., & Yashunin, D. A. (2020) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2018.2889473 · Cited by 1,904
Twilight zone of protein sequence alignments
Rost, B. (1999) · Protein Engineering, Design and Selection · doi:10.1093/protein/12.2.85 · Cited by 1,833 · cited 3×
“In order to intrude deeper into the twilight zone of sequence comparisons (Doolittle, 1986; Rost, 1999; Yona and Levitt, 2002), we need to fi nd a family of related proteins through pairwise alignments, compile a pro fi le or position-speci fi c scoring matrix (PSSM) from this family, and then use the…”
BackgroundSelf-citationUnified rational protein engineering with sequence-based deep representation learning
Alley, E. C., Khimulya, G., Biswas, S., AlQuraishi, M., & Church, G. M. (2019) · Nature Methods · doi:10.1038/s41592-019-0598-1 · Cited by 1,131 · cited 4×
“While the idea for this is not new (Alley et al., 2019; Littmann et al., 2021a; Littmann et al., 2021b; Ofer et al., 2021; Bileschi et al., 2022; Heinzinger et al., 2022; Nallapareddy et al., 2022), here we presented a more rigorous and generic framework for directly comparing embeddingbased to sequence-based alignmen…”
BackgroundInfluentialClustering huge protein sequence sets in linear time
Steinegger, M., & Söding, J. (2018) · Nature Communications · doi:10.1038/s41467-018-04964-5 · Cited by 875 · cited 4×
“…and have more free parameters than SeqVec (ProtBert 420 M — with M for million, ProtAlbert 224M, ProtT5 3B — with B for billion, vs. SeqVec 93 M) and were trained on the much larger BFD dataset (UniRef50 50 M) (Steinegger and Söding, 2018; Steinegger et al., 2019; Elnaggar et al., 2020).”
BackgroundMethodInfluentialSelf-citationEvaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 3×
“Embeddings from protein Language Models (pLMs) appear to carry information about aspects such as protein function, structure, and evolution (Rao et al., 2019; Littmann et al., 2020; Littmann et al., 2021a; Littmann et al., 2021b; Elnaggar et al., 2021; Marquet et al., 2021; Rives et al., 2021; Dunham et al., 2022; Hei…”
BackgroundModeling the language of life – Deep Learning Protein Sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · doi:10.1101/614313 · Cited by 606
Self-citationModeling aspects of the language of life through transfer-learning protein sequences
Heinzinger, M., Elnaggar, A., Wang, Y., Dallago, C., Nechaev, D., Matthes, F., & Rost, B. (2019) · BMC Bioinformatics · doi:10.1186/s12859-019-3220-8 · Cited by 606 · cited 10×
“ProtTrans (Elnaggar et al., 2021) abbreviates a collection of pLMs all of which use a transformer architecture and have more free parameters than SeqVec (ProtBert 420M – with M for million, ProtAlbert 224M, ProtT5 3B – with B for billion, vs. SeqVec 93M) and were trained on the much larger BFD dataset (UniRef50 50M) (…”
BackgroundMethodInfluentialSelf-citationProtein-level assembly increases protein sequence recovery from metagenomic samples manyfold
Steinegger, M., Mirdita, M., & Söding, J. (2019) · Nature Methods · doi:10.1038/s41592-019-0437-4 · Cited by 445 · cited 4×
“…and have more free parameters than SeqVec (ProtBert 420 M — with M for million, ProtAlbert 224M, ProtT5 3B — with B for billion, vs. SeqVec 93 M) and were trained on the much larger BFD dataset (UniRef50 50 M) (Steinegger and Söding, 2018; Steinegger et al., 2019; Elnaggar et al., 2020).”
BackgroundMethodInfluentialSelf-citationUsing deep learning to annotate the protein universe
Bileschi, M. L., Belanger, D., Bryant, D. H., Sanderson, T., Carter, B., Sculley, D., Bateman, A., DePristo, M. A., & Colwell, L. J. (2022) · Nature Biotechnology · doi:10.1038/s41587-021-01179-w · Cited by 332
Generalization through Memorization: Nearest Neighbor Language Models
Khandelwal, U., Levy, O., Jurafsky, D., Zettlemoyer, L., & Lewis, M. (2019) · arXiv · doi:10.48550/arxiv.1911.00172 · Cited by 285
Learning protein sequence embeddings using information from structure
Bepler, T., & Berger, B. (2019) · arXiv · doi:10.48550/arxiv.1902.08661 · Cited by 219 · cited 3×
“Some pLMs additionally include supervised training (Bepler and Berger, 2019; ElGebali et al., 2019).”
BackgroundProtein language model embeddings for fast, accurate, alignment-free protein structure prediction
Weissenow, K., Heinzinger, M., & Rost, B. (2021) · doi:10.1101/2021.07.31.454572 · Cited by 139
Self-citationLight attention predicts protein location from the language of life
Stärk, H., Dallago, C., Heinzinger, M., & Rost, B. (2021) · Bioinformatics Advances · doi:10.1093/bioadv/vbab035 · Cited by 130
Self-citationHigh-throughput deep learning variant effect prediction with Sequence UNET
Dunham, A. S., Beltrao, P., & AlQuraishi, M. (2023) · Genome Biology · doi:10.1186/s13059-023-02948-3 · Cited by 31 · cited 4×
“…about aspects such as protein function, structure, and evolution (Rao et al., 2019; Littmann et al., 2020; Elnaggar et al., 2021; Littmann et al., 2021a; Littmann et al., 2021b; Marquet et al., 2021; Rives et al., 2021; Dunham et al., 2022; Heinzinger et al., 2022; Weissenow et al., 2022).”
BackgroundInfluential
Cited by
Modeling conformational states of proteins with AlphaFold
Sala, D., Engelberger, F., Mchaourab, H. S., & Meiler, J. (2023) · Current Opinion in Structural Biology · doi:10.1016/j.sbi.2023.102645 · Cited by 182
Pantolini, L., Studer, G., Pereira, J., Durairaj, J., Tauriello, G., & Schwede, T. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btad786 · Cited by 55
Pantolini, L., Studer, G., Pereira, J., Durairaj, J., & Schwede, T. (2022) · doi:10.1101/2022.12.13.520313 · Cited by 55
Protein embedding based alignment
Iovino, B. G., & Ye, Y. (2024) · BMC Bioinformatics · doi:10.1186/s12859-024-05699-5 · Cited by 29
(2024) · doi:10.7554/elife.91415.3 · Cited by 3
Metadata
- Authors
- Schütze, K., Heinzinger, M., Steinegger, M., & Rost, B.
- Venue
- Frontiers in Bioinformatics
- Publisher
- Frontiers Media SA
- Published
- 2022-11-17
- Volume
- 2
- Pages
- 1033775-1033775
- ISSN
- 2673-7647 (ISSN-L)
- OpenAlex
- W4309609816
- Cited by
- 42
- References
- 31
- FWCI
- 2.28
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.