Deciphering the language of antibodies using self-supervised learning
Open accessThe DOI matches its Crossref record; 32 of the 32 references it cites were found in a registry or an index.
Abstract
An individual's B cell receptor (BCR) repertoire encodes information about past immune responses and potential for future disease protection. Deciphering the information stored in BCR sequence datasets will transform our understanding of disease and enable discovery of novel diagnostics and antibody therapeutics. A key challenge of BCR sequence analysis is the prediction of BCR properties from their amino acid sequence alone. Here, we present an antibody-specific language model, Antibody-specific Bidirectional Encoder Representation from Transformers (AntiBERTa), which provides a contextualized representation of BCR sequences. Following pre-training, we show that AntiBERTa embeddings capture biologically relevant information, generalizable to a range of applications. As a case study, we fine-tune AntiBERTa to predict paratope positions from an antibody sequence, outperforming public tools across multiple metrics. To our knowledge, AntiBERTa is the deepest protein-family-specific language model, providing a rich representation of BCRs. AntiBERTa embeddings are primed for multiple downstream tasks and can improve our understanding of the language of antibodies.
Overview
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Integrity signals
Registry
Cites
Statistics
Verification
Claims the paper makes
The claims and statistics of this paper haven’t been checked yet.
Free: it reads the open-access text and checks the first 5 claims that cite a source we can check. Create a free account to check every claim.
Self-check
Percentages, sums and reported differences re-computed from the paper’s own numbers.
The self-check needs the paper’s full text.
Citations
Cites
LeCun, Y., Bengio, Y., & Hinton, G. (2015) · Nature · doi:10.1038/nature14539 · Cited by 84,126
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2018) · arXiv · doi:10.48550/arxiv.1810.04805 · Cited by 73,173
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019) · Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) · doi:10.18653/v1/n19-1423 · Cited by 54,325
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., & Stoyanov, V. (2019) · arXiv · doi:10.48550/arxiv.1907.11692 · Cited by 16,515
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
Reimers, N., & Gurevych, I. (2019) · Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) · doi:10.18653/v1/d19-1410 · Cited by 12,778
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2019) · arXiv · doi:10.48550/arxiv.1910.10683 · Cited by 10,929
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
McInnes, L., Healy, J., & Melville, J. (2018) · arXiv · doi:10.48550/arxiv.1802.03426 · Cited by 7,101
Unsupervised Cross-lingual Representation Learning at Scale
Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2020) · Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics · doi:10.18653/v1/2020.acl-main.747 · Cited by 5,637
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2021) · Proceedings of the National Academy of Sciences · doi:10.1073/pnas.2016239118 · Cited by 3,286
Rives, A., Meier, J., Sercu, T., Goyal, S., Lin, Z., Liu, J., Guo, D., Ott, M., Zitnick, C. L., Ma, J., et al. (2019) · doi:10.1101/622803 · Cited by 3,280 · cited 9×
“To visualize the AntiBERTa embeddings, 1,000 BCR heavy chainswere randomly selected from a well-characterized public dataset(40) and then averaged over the length dimension before projection by uniform manifold approximation and projection (UMAP).(26,29,41) Despite only being given BCR sequences and no other informati…”
BackgroundMethodResultInfluentialProtTrans: Toward Understanding the Language of Life Through Self-Supervised Learning
Elnaggar, A., Heinzinger, M., Dallago, C., Rehawi, G., Wang, Y., Jones, L., Gibbs, T., Feher, T., Angerer, C., Steinegger, M., et al. (2022) · IEEE Transactions on Pattern Analysis and Machine Intelligence · doi:10.1109/tpami.2021.3095381 · Cited by 2,329 · cited 8×
“To visualize the AntiBERTa embeddings, 1,000 BCR heavy chainswere randomly selected from a well-characterized public dataset(40) and then averaged over the length dimension before projection by uniform manifold approximation and projection (UMAP).(26,29,41) Despite only being given BCR sequences and no other informati…”
BackgroundMethodInfluentialmT5: A Massively Multilingual Pre-trained Text-to-Text Transformer
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., & Raffel, C. (2021) · Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies · doi:10.18653/v1/2021.naacl-main.41 · Cited by 1,629
Evaluating Protein Transfer Learning with TAPE
Rao, R., Bhattacharya, N., Thomas, N., Duan, Y., Chen, X., Canny, J., Abbeel, P., & Song, Y. S. (2019) · doi:10.1101/676825 · Cited by 719 · cited 2×
“The 12-layer setup is well established as a strong baseline for protein language models and for several fine-tuning applications.(26,30) Furthermore, our comparisons to Sapiens provide a reasonable approximation on predictive performance at lower model depth.”
BackgroundMethodSAbDab: the structural antibody database
Dunbar, J., Krawczyk, K., Leem, J., Baker, T., Fuchs, A., Georges, G., Shi, J., & Deane, C. M. (2014) · Nucleic Acids Research · doi:10.1093/nar/gkt1043 · Cited by 674
Self-citationThe promise and challenge of high-throughput sequencing of the antibody repertoire
Georgiou, G., Ippolito, G. C., Beausang, J., Busse, C. E., Wardemann, H., & Quake, S. R. (2014) · Nature Biotechnology · doi:10.1038/nbt.2782 · Cited by 652 · cited 1×
“To recognize any potential antigen, an individual has a vast diversity of B cells with different B cell receptors (BCRs) – estimated to be as high as 1015 variants (1,2).”
BackgroundCommonality despite exceptional diversity in the baseline human antibody repertoire
Briney, B., Inderbitzin, A., Joyce, C., & Burton, D. R. (2019) · Nature · doi:10.1038/s41586-019-0879-y · Cited by 544
BERTology Meets Biology: Interpreting Attention in Protein Language Models
Vig, J., Madani, A., Varshney, L. R., Xiong, C., Socher, R., & Rajani, N. F. (2020) · doi:10.1101/2020.06.26.174417 · Cited by 338
Olsen, T. H., Boyles, F., & Deane, C. M. (2022) · Protein Science · doi:10.1002/pro.4205 · Cited by 319
Mason, D. M., Friedensohn, S., Weber, C. R., Jordi, C., Wagner, B., Meng, S. M., Ehling, R. A., Bonati, L., Dahinden, J., Gainza, P., et al. (2021) · Nature Biomedical Engineering · doi:10.1038/s41551-021-00699-9 · Cited by 308
High frequency of shared clonotypes in human B cell receptor repertoires
Soto, C., Bombardi, R. G., Branchizio, A., Kose, N., Matta, P., Sevy, A. M., Sinkovits, R. S., Gilchuk, P., Finn, J. A., & Crowe, J. E. (2019) · Nature · doi:10.1038/s41586-019-0934-8 · Cited by 294 · cited 1×
“Most analyses have focused on comparing high-level differences in aggregate BCR repertoire metrics between cohorts, such as differences in diversity, number of somatic hypermutations,(8) isotype subclass usage, and V(D)J gene segment usage.(9,10) To realize the full potential of the data, it will be necessary to under…”
BackgroundKovaltsuk, A., Leem, J., Kelm, S., Snowden, J., Deane, C. M., & Krawczyk, K. (2018) · The Journal of Immunology · doi:10.4049/jimmunol.1800708 · Cited by 247
Self-citationAntibody structure prediction using interpretable deep learning
Ruffolo, J. A., Sulam, J., & Gray, J. J. (2022) · Patterns · doi:10.1016/j.patter.2021.100406 · Cited by 216 · cited 1×
“While we focus on paratope prediction here, the AntiBERTa model can potentially be fine-tuned for other tasks, such as antibody structure prediction and humanization.(13,36) For each position in the antibody sequence, we predict the probability that it is part of the antibody’s paratope (see experimental procedures).”
MethodBioinformatic and Statistical Analysis of Adaptive Immune Repertoires
Greiff, V., Miho, E., Menzel, U., & Reddy, S. T. (2015) · Trends in Immunology · doi:10.1016/j.it.2015.09.006 · Cited by 210 · cited 1×
“Most analyses have focussed on comparing high-level differences in total BCR repertoire metrics between cohorts, such as differences in diversity, number of somatic hypermutations (8), isotype subclass usage, and V(D)J gene segment usage (9,10).”
BackgroundPopovic, B., Breed, J., Rees, D. G., Gardener, M. J., Vinall, L. M. K., Kemp, B., Spooner, J., Keen, J., Minter, R., Uddin, F., et al. (2017) · Journal of Molecular Biology · doi:10.1016/j.jmb.2016.12.005 · Cited by 198 · cited 1×
“Likewise, for the light chain of tralukinomab (PDB: 5L6Y; (51)), another example in our test set, AntiBERTa correctly predicts paratope positions outside of the CDRs (Figure 4C, D).”
BackgroundPrihoda, D., Maamary, J., Waight, A., Juan, V., Fayadat-Dilman, L., Svozil, D., & Bitton, D. A. (2022) · mAbs · doi:10.1080/19420862.2021.2020203 · Cited by 190
Parapred: antibody paratope prediction using convolutional and recurrent neural networks
Liberis, E., Veličković, P., Sormanni, P., Vendruscolo, M., & Liò, P. (2018) · Bioinformatics · doi:10.1093/bioinformatics/bty305 · Cited by 165
ABlooper: fast accurate antibody CDR loop structure prediction with accuracy estimation
Abanades, B., Georges, G., Bujotzek, A., & Deane, C. M. (2022) · Bioinformatics · doi:10.1093/bioinformatics/btac016 · Cited by 157
Humanization of antibodies using a machine learning approach on large-scale repertoire data
Marks, C., Hummer, A. M., Chin, M., & Deane, C. M. (2021) · Bioinformatics · doi:10.1093/bioinformatics/btab434 · Cited by 124 · cited 1×
“We then used AntiBERTa and ProtBERT to embed the heavy chains from 198 well-characterized therapeutic antibodies.(42,43) AntiBERTa is generally able to separate therapeutic antibodies according to their origin (i.”
MethodUnderstanding the human antibody repertoire
Rees, A. R. (2020) · mAbs · doi:10.1080/19420862.2020.1729683 · Cited by 108 · cited 1×
“To recognize any potential antigen, an individual has a vast diversity of B cells with different B cell receptors (BCRs)—estimated to be as high as 10(15) variants.(1,2) BCR repertoire diversity is generated through the process of somatic recombination of V, D (heavy chain only), and J gene segments during B cell deve…”
BackgroundLapidoth, G. D., Baran, D., Pszolla, G. M., Norn, C., Alon, A., Tyka, M. D., & Fleishman, S. J. (2015) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.24779 · Cited by 107
The H3 loop of antibodies shows unique structural characteristics
Regep, C., Georges, G., Shi, J., Popovic, B., & Deane, C. M. (2017) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25291 · Cited by 99
Antibody side chain conformations are position-dependent
Leem, J., Georges, G., Shi, J., & Deane, C. M. (2018) · Proteins Structure Function and Bioinformatics · doi:10.1002/prot.25453 · Cited by 22
Cited by
ProGen2: Exploring the boundaries of protein language models
Nijkamp, E., Ruffolo, J. A., Weinstein, E. N., Naik, N., & Madani, A. (2023) · Cell Systems · doi:10.1016/j.cels.2023.10.002 · Cited by 480
Fast, accurate antibody structure prediction from deep learning on massive set of natural antibodies
Ruffolo, J. A., Chu, L.-S., Mahajan, S. P., & Gray, J. J. (2023) · Nature Communications · doi:10.1038/s41467-023-38063-x · Cited by 316
Fine-tuning protein language models boosts predictions across diverse tasks
Schmirler, R., Heinzinger, M., & Rost, B. (2024) · Nature Communications · doi:10.1038/s41467-024-51844-2 · Cited by 229
“I understand that the tasks are very diverse, but the sense that metrics like Precision and Recall give, where you see the trade-off, is hard to get from Spearman correlation from 0.72 to 0.736 for instance.”
InfluentialIn silico proof of principle of machine learning-based antibody design at unconstrained scale
Akbar, R., Robert, P. A., Weber, C. R., Widrich, M., Frank, R., Pavlović, M., Scheffer, L., Chernigovskaya, M., Snapkov, I., Slabodkin, A., et al. (2022) · mAbs · doi:10.1080/19420862.2022.2031482 · Cited by 94
Prediction of antibiotic resistance mechanisms using a protein language model
Yagimoto, K., Hosoda, S., Sato, M., & Hamada, M. (2024) · Bioinformatics · doi:10.1093/bioinformatics/btae550 · Cited by 25
Metadata
- Authors
- Leem, J., Mitchell, L. S., Farmery, J. H. R., Barton, J., & Galson, J. D.
- Venue
- Patterns
- Publisher
- Elsevier BV
- Published
- 2022-07
- Volume
- 3
- Issue
- 7
- Pages
- 100513
- ISSN
- 2666-3899 (ISSN-L)
- OpenAlex
- W4280625391
- Cited by
- 154
- References
- 32
- FWCI
- 8.56
- Type
- journal article
- Language
- en
- Open access
- gold
About these checks
Registry, retraction, Cites and statistics results are computed, not judged, with one exception: a printed reference that no index lists is read once by a model before it counts as a citation error. Claims are extracted by a model and kept only when their quote appears verbatim in the text; each cited source is then checked, with a model’s help, for whether it says what the claim relies on. Those verdicts can be wrong — read the claim’s quote and the cited source before relying on one.
Statistics and arithmetic are recomputed, not judged.
Confidence bands are shown only for fields calibrated on our annotated test set. None is calibrated yet, so no band is shown.
What counts as a claim ›How we measure confidence ›How we check this ›
Are you an author, or do you see a problem with this page? Request a review
Data sources
Bibliographic data: Semantic Scholar, OpenAlex, Crossref.
Contains information from the Semantic Scholar Academic Graph Datasets, made available under the ODC Attribution License (ODC-BY 1.0).
Semantic Scholar’s data is described in: Kinney, R., et al. (2023). The Semantic Scholar Open Data Platform. arXiv:2301.10140.